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RAISING QUALITY OF PHYSICS EDUCATION: CONTRIBUTION OF JBSE OVER THE 
PAST ISSUES 


Peter Demkanin 
Comenius University, Slovakia 


| have worked more than thirty years in physics education (as some of the readers of this journal), many 
years as a secondary school physics teacher in various curricula (as some of the readers), and more than 20 years 
at university doing research and preparing future physics teachers (as some of readers). | am also the author of 
a physics textbook for secondary education, and now | am working on a new one. Naturally, | would like to have 
the new textbook, like my other outputs, based on the current state of knowledge and well-developed and well- 
applied theories behind physics education. And here is the seed of the question - how does our journal contribute 
to raising the quality of our outputs — in my case, increasing the quality of physics education? | first look at the 
goals of physics education. Here, | use the most straightforward taxonomy of goals presented in (Demkanin 2013): 
goals related to attitudes of society to science, goals related to methods of science, and goals related to particular 
knowledge. The last one | split into two sub-goals — knowledge selected to develop methods and attitudes and 
knowledge selected to raise the quality of living and general scientific culture. Of course, the goals we can reach 
by means - by the methods of education — methods of teaching and learning. So, let's look at a few previous issues 
of JBSE and at the contributions having the potential to raise the quality of physics education. | have mentioned 
only some of the contributions | will probably use in the next few years. | tried to focus on physics education, not 
explicit chemistry or biology education, even if some of such articles could be fully relevant to my work. 


Goals Related to Attitudes of Society to Science 


Lamanauskas (2022) emphasised that the natural science literacy of society is insufficient, and its relevance 
significantly increased in the last decades. In the previous issues of JBSE, we clearly see the importance of environ- 
mental education — influencing public opinion about climate change and environmental issues. Lavonen (2022), 
discussing climate education, highlighted the need for well-educated physics teachers and a new paradigm for 
preparing physics teachers. AberSek (2021) discussed the gap between scientists, pseudo-scientists, influencers and 
politicians, and this gap illustrates the topic of global warming and other ecological issues. Agostinis-Sobrino et al. 
(2021) discussed health education and lifestyle behaviour, which also could be relevant to changing attitudes to 
physics education. An interesting term, The Happy Planet Index, is discussed in Janouskova & Bilek, 2022). Analysis 
of school textbooks on Physics and other natural science subjects is offered by Revak et al. (2023), who analysed 
energy awareness education in these books. 
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Goals Related to Methods of Science 


Wider and Wider (2023) examined students’ problem-solving skills and related metacognitive abilities. Taking 
dimensions such as monitoring, regulation, and evaluation (metacognitive), understanding the problem, devis- 
ing a plan, carrying about the plan, and looking back, they looked deeply at students’ problem-solving skills. As 
formal education, physics education is rooted in natural science education in primary schools. Lamanauskas (2022) 
highlighted the need to solve educational problems relevant to primary school pupils, forming their life values. 
In his research, he found that discussions are not encouraged enough in Lithuania at the level of primary schools. 
Much more popular are practical activities and demonstrations. Improving science process skills of students with 
mild intellectual disabilities is examined by Senel Coruhlu et al. (2023). 


Goals Related to Knowledge Selected for the Development of Methods and Attitudes 


lam sure we have some articles dealing with physics knowledge selected for school courses to develop sci- 
ence process skills and attitudes to science, but | did not find anything explicit in the past issues. One more, this 
does not mean that we do not have such articles. Some readers can argue that everything we teach is developing 
methods of science and attitudes. 


Goals Related to Knowledge Selected to Raise the Quality of Living and General Scientific Culture 


The concept of inertia, an essential topic of mechanics, with some critical notes from the history of this topic, is 
discussed by Bussotti (2021). Alternative conceptions of students in Newtonian mechanics are examined by Bahtaji 
(2023). Learning heat and temperature concepts by pupils at primary schools are discussed by Tseng et al. (2023). 


Teaching Physics 


In the context of climate education, Lavonen (2022) highlighted the relevance of project-based teaching 
methods, guiding students to ask relevant questions related to the complex phenomenon, critically search for 
information, participatory approaches, and affect-driven approaches. Alp and Coskun Onan (2023) analysed the 
possibilities for using comics to constitute climate change awareness in 5" grade. Abersek (2021) highlighted, in 
the context of science versus pseudo-science, the development of critical thinking and strategic separation of truth 
(ideas based on science) from potential lies (alternative ideas contrary to scientific ideas, myths). The development 
of the physics module applying categories such as personal interest, sense-making, and effort is discussed by 
Sulaiman et al. (2023). The difference between end-of-chapter problems and inquiry activities used as homework 
was investigated by Simic et al. (2023), who found that simulation-based and video-based homework contributes 
to the improvement of conceptual understanding in mechanics. The role of multimedia (simulations, animations, 
videos, models) in teaching university students black body radiation, photoelectric effect and other quantum 
physics topics is discussed by Nyirahabimana et al. (2023). Interesting factors related to the social responsibility of 
scientists and engineers, as viewed by pre-service physics teachers, are analysed by Zhang (2023). The typology 
of questions used by science teachers is examined by Saka and Inaltekin (2023). 


Learning Physics 


Illustrated on science teacher education, Feser (2021) discusses the topic related to future science teacher 
learning, the topic of belonging to science. How do physics teachers feel as an integral part of the physics com- 
munity? How do physics researchers feel physics teachers belong to the physics community? Secondary school 
students’ perception of the science learning environment, especially metacognition and self-efficacy, is analysed 
by Kim and Alghamdi (2023). A construal level theory, which includes the term psychological distance, illustrates 
environmental issues Abersek (2022). Toulmin’s argument pattern and its impact on the learning of lower sec- 
ondary school pupils in the context of the solar system, force and motion, and systems in the human body were 
analysed by Acar and Azakli (2023). They report that using this argument pattern raised pupils’ metacognition and 
logical thinking. The effect of a particular combination of home and classroom activities encouraging students 
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to take more responsibility for their learning is examined in the context of direct electric current topics Radulovi¢c 
et al. (2023). Physics teachers’ learning how to use research papers to creatively develop teaching sequences of a 
particular quality (based on the information from the research papers) was studied by (Park, Yoon, Lee, 2023). Our 
journal, as well as general physics education, includes the education of students with mild intellectual disabilities. 
Improving their science process skills was examined by Senel Coruhlu et al. (2023). 


Contribution to the Raising Quality of Future Physics Education 


Empirical research well set to a theoretical background, without doubt, is much better than empirical research 
based only on the intuition of a person with limited experience. Even if | have more than 30 years of experience 
with secondary school students and university students — students in pre-service studies, in-service courses and 
students of hard science, | still feel a need to cultivate my theoretical background. One type of activity related to 
this need is looking at JBSE articles, letting them go through the filter of my previous experiences and theoretical 
knowledge, allowing them to resonate, interfere, constructively or destructively. It is quite a pleasant experience 
to identify a myth | have used as a truth for years. Similarly, it is a pleasant experience to see that some groups of 
people thousands of kilometres from my hometown, Bratislava, think likewise and believe in the same principles 
and tenets of physics education as | use in my research and teaching practice. 

Physics, Physics Education, and the Theory of Physics Education are often considered quite old science disci- 
plines. Some say that the roots are in Newton's Principia (Newton's laws are probably in every physics textbook), 
some say that even Archimedes was important in the development of Physics (Archimedes’ principle is perhaps in 
every physics textbook), methods of education applied to teaching Newton's laws and Archimedes principle are 
probably in every course on didactics of physics. Where to go in raising the quality of physics education? Better 
selection of physics topics taught in schools — to cultivate attitudes of society, better use of new digital technolo- 
gies — to cultivate the abilities to use methods of science? Let me be less abstract — let’s look at concrete material - 
water/ice and graph its density versus temperature (in stable states), Figure 1, and detail in Figure 2. 


Figure 1 
Density-Temperature Graph for Illustration of Context 
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Figure 2 


Interesting Detail on the Density-Temperature Graph, Note the Unit of Temperature Used 
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What to teach and how to teach about water and ice? How can we raise the quality of using contexts related 
to water and ice in physics education? And how to teach? And how do our students learn? 

One way that naturally emerged in the last decades could be based on the application of new knowledge in 
neurosciences, in the science of mind-brain-education. Similarly, as physics education was influenced by research- 
ers such as Piaget, Maslow, and Bruner in the second half of the 20th century, nowadays, we can try to incorporate 
into our efforts the results of research of groups focusing on possibilities of applying knowledge based on new 
opportunities of brain scanning, artificial intelligence. Besides articles published in JBSE, | often look at some books 
and can recommend one here. Tokuhama-Espinosa (2021), in her book Bringing the Neuroscience of Learning to 
Online Teaching, brings many ideas applicable to the theoretical backgrounds of our articles related to improving 
physics education in the near future. 
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MICROBIOLOGICAL AWARENESS 
AMONG UPPER-SECONDARY 
SCHOOL STUDENTS IN 

THE CONTEXT OF COVID-19 
VACCINATION 


Beniamin Abramczyk, 
Szymon tawicki, 
Weronika Pyter, 
Agata Bluszcz, 

Ignacy Piszczek, 
Jonatan Audycki, 
Julia Pawtowska 


Introduction 


Vaccination is one of the most efficient and cost-effective procedures 
for preventing hospitalization and death from diseases, including COVID-19. 
Vaccines also play a major role in preventing the transmission of the disease 
and a modest one in the case of COVID-19 (Kraaijeveld, 2022). Although 
children and adolescents are less liable to the severe cause of this disease 
and less likely to die as a result of it (Viner et al., 2021), researchers empha- 
size the high value of vaccinating young people as they can transmit the 
infection to people at high risk - the elderly or chronically ill. Especially if 
one considers that young people lead a more intense social life and thus 
come into contact with more people - including potential carriers of the 
disease (Zimet et al., 2020). 

The 2009 experience of the influenza epidemic (H1N1 virus) shows that 
societies are reluctant to vaccination - rates of vaccination-friendly attitudes 
in individual countries ranged from 17% to 67% (Al-Mistarehi et al., 2021). The 
more recent pandemic of COVID-19 seems to have demonstrated a similar 
tendency when the general public’s cold reception of the developed vaccines 
challenged public health. Interestingly, data on attitudes toward vaccina- 
tion among adults indicate that a higher level of education correlates with 
amore favorable attitude toward vaccination (Al-Mistarehi et al., 2021; Afifi 
et al., 2021; Lin et al., 2020), which is also true in Poland as demonstrated by 
Gotebiowska et al. (2023), and Raciborski et al. (2021). Moreover, the stream 
of studies also influences the decision whether to get vaccinated or not. For 
example, Sorgo et al. (2022) demonstrated that Slovenian medical students 
are more inclined to get vaccinated against COVID-19 than their peers who 
study different fields, including health care related subjects. 

Nevertheless, education is not the only factor that impacts vaccination 
willingness in adults. Other factors such as place of residence (Raciborski et 
al., 2021; Gotebiowska et al., 2023), lifestyle (Fazel et al., 2021), and being a 
medical practitioner (Wang et al., 2020) have also been proven to influence 
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the attitude towards vaccinating. It has been found that medical students favor vaccination more than their peers 
who attend non-medical degree courses. According to an analysis carried out by Knubchandani et al. (2022) in 39 
countries, COVID-19 vaccinations are strongly rejected by an average of 18.9% of medical students and as many 
as 22% of non-medical students. 

To date, few studies have examined a big population of adolescents in the context of their vaccination will- 
ingness. In Austria, the impact of the type of school was studied - it was shown that students of general upper 
secondary schools are much more favorable to vaccination than those participating in vocational education (53% 
of applicants and an indicator of 4/5 points vs. 28% of applicants and an indicator of 3.16 /5 points - where 5 is “I 
definitely want to get vaccinated” and 1 is “I definitely don’t want to get vaccinated” (Humer et al., 2021)). Other 
questionnaires conducted among adolescents from different countries indicate that slightly more than half of the 
respondents declare the willingness to be vaccinated, for example: 

e inthe USA, age 12-17: 64% (26.1% vaccinated, 38.2% willing to get vaccinated) (Scherer et al., 2021); 
e Canada, age 16-21: 65.4% (Afifi et al., 2021). 
e Austria, upper secondary schools 53% (Al-Mistarehi et al., 2021); 

Research from Great Britain shows that the older the students, the more they favor vaccination, and fewer are 
hesitant. On the other hand, the percentage of opponents is relatively constant (Fazel et al., 2021). 

When examining education as a factor influencing vaccination willingness, one should also consider signifi- 
cant changes in the Polish education system, which happened several times in the last 20 years. The extensive 
structural reform of 1999 increased the autonomy of the schools and teachers, introduced standardized testing, 
but most importantly, cut the primary school from 8 to 6 years and established lower secondary schools (called 
gymnasiums) that children were obliged to attend before upper secondary school (Jakubowski, 2020; Wisniewski 
& Zahorska, 2020). The reform led to a significant improvement in Polish students’ PISA performance. The aver- 
age Polish student's results have improved by over 30 points in reading since 2000 and by about 25 points in 
mathematics since 2003. Poland experienced the most considerable improvement from 2000 to 2018 among the 
OECD countries (Jakubowski, 2020). Despite that, in 2017, another major structural reform that canceled lower 
secondary schools was introduced. The reform was rapidly implemented without much thought, preparation, or 
consultation with the teachers in the name of ‘the good old days’ (Wisniewski & Zahorska, 2020). It is important to 
note that the most recent PISA results are from 2018 and therefore are a product of children educated in a system 
introduced by the 1999 reform. 

Vaccination willingness was also studied in the light of one other factor than education level among school 
children. The influence of the socioeconomic situation (including migration history) and gender was established 
on the students’ approach and their guardians’ (Humer et al., 2021; Scherer et al., 2021). 


Research Problem 


It is expected that education has a major influence on attitudes toward vaccination. However, there are no 
studies on schoolchildren that would directly link the scope of their biological knowledge with the willingness 
to vaccinate against COVID-19. The relationship, or lack thereof, is especially interesting since most conventional 
schools mainly focus on transferring and examining general theoretical knowledge. 

Furthermore, the influence of other factors (place of residence, lifestyle, and close acquaintance with a 
medical practitioner) on willingness to vaccinate against COVID-19 was never studied in upper secondary school 
students in Poland. Even though in the research performed by Gotebiowska et al. (2023) some representatives of 
this group were probably present, they constituted less than 2.5% of the studied population and their answers 
were not analyzed separately. 


Research Aim 


The aim of the present research was to determine whether class profile, place of residence, lifestyle, and close 
acquaintance with a medical professional influence microbiological awareness and the decision to get vaccinated 
against COVID-19 among high school students. A question of whether the level of microbiological awareness 
impacts one’s willingness to be vaccinated against COVID-19 was also addressed. In this case, microbiological 
awareness was understood as basic and strictly pragmatic microbiology knowledge that a Polish high school 
attendee should possess according to the official curriculum. The study was conducted in Poland, and, in this 
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context, it is important to note that in the Polish education system, high school students can learn each subject 
on two levels - basic and extended. The basic level is more pragmatic and obligatory for all students, while the 
extended is optional and is meant to prepare students for an extended level exam considering a particular subject 
(The Ministry of National Education, 2018). Consequently, it was presumed that learning a subject in an extended 
scope could affect microbiological awareness. 


Research Methodology 
General Background 


A survey strategy was implemented for this research with the use of an online questionnaire (Table $1). The 
questionnaire aimed to gather information about microbiological awareness of students and their vaccination 
willingness. The answers were collected between November 2021 and April 2022 from students attending upper 
secondary schools across Poland. 


Participants 


The questionnaire was shared on social media. However, only several students submitted their answers. It 
was decided to recruit participants by emailing information about the possible study enrollment to the princi- 
pal's office. The authors’ Python script was used to send the emails to all high schools located in 6 voivodeships 
(Masovian, Greater Poland, Warmian-Masurian, Silesian, and Podlaskie) listed in the Polish Registry of Schools and 
Educational Facilities (https://rspo.gov.pl/). Unfortunately, only some headmasters agreed and expressed the will 
for their students to participate. All of the schools who agreed were enrolled in the study. 

Consequently, the questionnaire was filled by 1215 respondents from 6 abovementioned voivodeships: 3 
schools from Masovian, 1 school from Greater Poland, 1 school from Warmian-Masurian, 2 schools from Silesian, 
and 2 schools from Podlaskie. These voivodeships create a nearly diagonal section through Poland. Therefore, the 
data were recognized to be fairly representative. However, low responsiveness of schools prevented the authors 
from obtaining more answers. 

In further analysis, 1170 responses were considered. 45 responses were not taken into account due to (a) im- 
perfection of the constructed questionnaire - problems with grading the answer“! don’t know this rule”in question 
7 - see Supplementary Table 1 (43 responses), (b) vulgar answers in open questions indicating that the participant 
did not take the questionnaire seriously (2 responses). 

The considered respondents declared to study varied subjects on the extended level: 693 of them studied 
biology on the extended level, 605 - chemistry, 528 - foreign language, 394 - mathematics, 235 - Polish, 190 - his- 
tory, 148 - social sciences (“wiedza o spoteczenstwie”), 147 - geography, 124 - physics. 888 respondents declared 
to be vaccinated against COVID-19, 59 were not but planned to be in the future, 123 were not vaccinated and were 
hesitant to get the vaccination, 100 were not vaccinated and refused ever to be vaccinated against COVID-19. 


Instrument and Procedures 


An online questionnaire (Table $1) in Google Forms was designed. The questionnaire consisted of a contextual 
part and a cognitive part. In the contextual part, students were asked about their grade, place of residence, and 
if one or more of their family members have work experience in medicinal sciences (e.g., is/was a doctor, nurse, 
technician, etc.). Participants were also asked about extended subjects and preferred ways of spending free time. 
Based on this question, students were assigned to one of four groups: physically active, socially active, physically 
and socially active, or inactive. 

The cognitive part focused on knowledge of microbiology. Respondents answered questions about basic 
microbiological knowledge, infectious disease prevention, and attitude towards vaccination against COVID-19. 
All questions in the cognitive part were constructed based on the Polish biology curriculum (Table $2), ensuring 
their validity. Questions and their aim are explained in Table $1. 

Participants earned points for correct answers in the microbiological awareness part of the questionnaire. 
The maximum number of points in each question was equal to the number of correct answers in that question. In 
questions in which more than one answer was correct, points were subtracted for each incorrect answer, and the 
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minimum number of points participants could earn for those questions was zero. It was assumed that the number 
of earned points was proportional to the microbiological awareness of the participants. 

The consent for students’ participation in the study was sought with the school’s headmaster. The principal 
was informed about the aims of the study in detail and was given an opportunity to examine the questionnaire. 
Afterwards, the questionnaire was sent by the headmaster electronically to the students. Participation in the sur- 
vey was voluntary. The questionnaire contained information about the aims of the study so the students’ consent 
to participate was informed. Participants could quit the study at any moment. No headmasters nor teachers had 
access to the results of the survey. The questionnaire did not collect personal data as understood in the General 
Data Protection Regulation (Regulation (EU) 2016/679) nor the information about the school to which a participant 
attended. 


Reliability of the Instrument 


The reliability of the cognitive part was assessed by calculating the w coefficient, which was proved to be 
the most reliable (Zinbarg et al., 2005). Corrected item-total correlations (Cureton, 1966; v2.3.6; Revelle, 2023) and 
discrimination indexes (biserial point correlations) were also calculated for each question. To assess the difficulty 
of tasks, response frequency scores (defined as the mean score divided by the maximum possible score) were 
calculated (Brzezinska, 2020). 

The calculated w coefficients equaled: w hierarchical = .44, w total = .59, and w hierarchical asymptotic = .75. 
It was decided that the asymptotic version of the w hierarchical coefficient will be considered because it produces 
the least biased results of general factor reliability (Trizano-Hermosilla et al., 2021). Consequently, it was concluded 
that the reliability of the test was satisfactory. 

The corrected item-total correlations exceeded .2 for all tasks, showing their acceptable consistency with 
the rest of the instrument (Collins et al., 2010). The discrimination index of five questions was higher than .4, indi- 
cating good discrimination. For three other tasks, the discrimination index was above .2, which was qualified as 
fairly discriminating (McGahee & Ball, 2009). Question 2 was recognized as a task of the lowest quality showing a 
discrimination index lower than .2 and a corrected item correlation of .206. However, this question referred to an 
important topic of human symbiotic microbiota. Consequently, it was decided not to remove it from the analysis. 

Question 2 was also recognized as the easiest task, with a response frequency score of .979. On the contrary, 
Question 3 was considered the hardest. Results for all items analyzed are presented in Table 1. 


Table 1 
Results of the Quality Analysis for Each Question Asked In the Cognitive Part of the Questionnaire 


Question Corrected item total correlation Discrimination index Response frequency score 
Question 1 412 515 .385 
Question 2 .206 .187 .979 
Question 3 504 133 274 
Question 4 366 565 524 
Question 5 .284 505 516 
Question 6 .250 241 .910 
Question 7 253 .266 864 
Question 8 356 443 423 
Question 9 295 367 315 
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Data Analysis 


Most of the respondents chose more than one extended subject. Consequently, to determine the influence 
of participants’ chosen extended subject on microbiological awareness, groups of respondents that chose a 
particular extended subject and all participants that did not were compared. This analysis was conducted for the 
following extended subjects: social studies, foreign language, Polish language, mathematics, history, geography, 
physics, chemistry, and biology. Microbiological awareness of participants who were related to a medical worker 
and those who were not were compared. Respondents who answered “I don’t know” in the questionnaire were 
grouped with those who answered “No.’ It was assumed that since the participants do not know if their relatives 
work in the medical field, it would not significantly affect them. For all groups of data Shapiro-Wilk’s normality tests 
were conducted. In all cases, data could not be approximated by the normal distribution. 

Consequently, the Kruskal-Wallis test was used. Effect sizes were measured by calculating the €? coefficient 
for data showing statistically significant differences. Means (M) and standard deviations (SD) of obtained points 
within groups were calculated. 

Microbiological awareness differences depending on the place of living were studied. Because of the low 
number of participants living in cities ranging from 50,000 to 150,000 citizens and 150,000 to 500,000 citizens, 
they were grouped into a single category of cities ranging from 50,000 to 500,000 citizens. Four groups of results 
were obtained: participants who lived in villages, towns smaller than 50,000 citizens, cities ranging from 50,000 
to 500,000 citizens, and cities larger than 500,000 citizens. Microbiological awareness differences were examined 
depending on participants’ vaccination and physical activity decisions. Based on the answers about their physical 
activity, participants were grouped into four categories: 

a) Physically and socially active (spend time both on sports and activities with friends and family), 
b) Physically active (spend time on sports, but not on activities with friends and family), 

c) Socially active (spend time on activities with friends and family, but not on sports), 

d) Not active (do not spend time on sports or activities with friends and family) 

For this analysis, the Kruskal-Wallis test was used. If the differences between groups were statistically sig- 
nificant, post-hoc Dunn's test was performed. Means (M) and standard deviations (SD) of obtained points within 
groups were calculated. 

It was studied if willingness to vaccinate depends on the place of living. Two groups were distinguished: par- 
ticipants living in villages and towns of less than 50,000 inhabitants and from cities of more than 50,000 citizens. 
It was also examined if willingness to vaccinate depends on physical activity (groups a, b, and c were merged) and 
close relationships with medical workers. Chi-squared tests were conducted, including described categories and 
participants’ willingness to vaccinate. Theoretical distributions were calculated for the datasets in which differ- 
ences were statistically significant. Effect sizes were measured by calculating the @ coefficient for data showing 
statistically significant differences. 

Statistical analysis was conducted using the R software (R Core Team, 2013), dunn. test package (Dinno & 
Dinno, 2017), psych package (v2.3.6; Revelle, 2023), and rcompanion package (v2.4.30; Mangiafico, 2023) according 
to the procedures described. Statistical significance at p < .05 was assumed. €? Values were interpreted as follows: 
& >.14= strong effect, .14 > €? > .06 = medium effect, .06 > €? > .01 = weak effect, €? < .01 = no effect (Khalilzadeh 
&Tasci, 2017). p Coefficient values were interpreted as follows: @ > .5 = strong effect, .5 > @ > .3 = medium effect, 
.3>@Q>.1=weak effect, p < .1 = no effect (Cohen, 1977). 


Research Results 
Microbiological Awareness 
On average, the respondents scored 8.5 points (out of 17 points possible in the test) with a standard devia- 


tion of SD = 2.7. The maximum score obtained was 16. The minimum score obtained equaled 0. The frequency 
distribution of all scores is presented in Figure 1. 
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Figure 1 


Histogram of Number of Points Obtained In the Knowledge Test 
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Note. The height of a bar indicates the number of students who obtained a particular number of points in the cognitive part of 
the questionnaire. 


Extended subjects that had a moderate positive influence on microbiological awareness were biology (p < .001, 
€? = .0837) and chemistry (p < .001, €? = .0912). It should be mentioned that only 11 people studying chemistry did 
not study biology at the extended level. Therefore, it was impossible to examine the influence of studying chemistry 
without the obvious influence of studying biology simultaneously. Mathematics, without excluding individuals 
studying biology on the extended level, seemed to also have a positive impact (p = .003). However, the €” value of 
.0072 indicated no significant effect. 

Extended subjects that had a weak negative influence on microbiological awareness were geography (p < .001, 
€? = 0239), history (p < .001, €* = .0207), Polish (p < .001, €? = .0234), social studies (p < .001, €? = .0227) and math- 
ematics when individuals studying biology were excluded (p < .001, €? = .0148). Studying physics also seemed to 
impact microbiological awareness negatively (p = .006); however, a low €? (.0064) indicated no effect. 

In the case of foreign languages, no significant differences were detected in the total sum of points scored on 
the test in the case of people who studied this subject and those who did not take it at an extended level (p = .973). 
All of the above results are presented in Figure 2. The exact average score values and standard deviations for each 
subject are presented in Table S3. 
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Figure 2 
The Influence of Studying Different Subjects at an Extended Level on the Microbiological Awareness of Upper-Secondary 


School Students 
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Note. The height of the bars indicates the mean number of points obtained in the test. Error bars indicate standard deviation in 
designated groups. that study the subject at an extended level (YES) compared to those that do not (NO) Statistically significant 
differences between the groups is marked with solid line patterns (p < .05). Statistically insignificant differences between the 
groups are marked with dotted line patterns. €? > .01 for the designated data groups is marked with asterisks. “*mathematics” 
indicates students who study mathematics at an extended level but not biology. 


Significant differences were detected in the number of points obtained by people with a medical professional 
among their relatives compared to those who do not have such a person (p = .003). Students with a medical rela- 
tive performed slightly better - M = 8.7 (SD = 2.8) versus M = 8.3 (SD = 2.6). However, no effect of relative health 
professionals on microbiological awareness was observed (€? = .0073). 

Statistical significance was not achieved in the case of differences in the number of points obtained depend- 
ing on the place of residence (p =.11, €? = .0051). 

The number of points obtained differed significantly depending on the willingness to get vaccinated (p =.012). 
Students who were vaccinated or expressed the will to be obtained an average of 8.6 points (SD = 2.8). Those who 
were hesitant or strongly refused ever to be vaccinated against COVID-19 obtained an average of 8.0 points (SD = 
2.7). However, no effect of willingness to get vaccinated on microbiological awareness was observed (€? = .0054). 

There were significant differences in the number of points obtained depending on the respondents lifestyle (p 
= .008). After more detailed analysis, it turned out that the groups that showed significant differences were “physi- 
cally and socially active” and “not active” (p = .010 - obtained from Dunn's test). On average, physically and socially 
active students obtained a higher score on the test (M = 8.8; SD = 2.7) than non-active students (M = 8.1; SD = 2.7). 
However, no effect of lifestyle on microbiological awareness was observed (€? = .0095). For the remaining groups, 
no significant differences were found. All of the above results are presented in Figure 3. 
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The Influence of Different Factors on the Microbiological Awareness of Upper-Secondary School Students 
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Note. The height of the bars indicates the average number of points obtained in the test. Error bars indicate standard deviation 
in designated groups. In the section “Willing to be vaccinated,” students included in the group “YES” were vaccinated or planned 
to at the time of taking the questionnaire (low-density stripes). Students included in the group “NO” were hesitant or refused 
ever to be vaccinated against COVID-19 (high-density stripes). In the “Place of residence” section, the citizen number (thousands) 
of towns and cities where respondents lived is indicated. Rising citizen numbers of place of residence are indicated by increasing 
pattern density. In the section “Lifestyle,” designated groups are “physically and socially active” (A), “physically active” (B), “socially 
active” (C), and “not active” (D). The decreasing activeness of designated groups is indicated by the rising density of the pattern. 
The statistical significance of differences between groups is marked with solid lines within the bars (p < .05). Dotted lines within 
the bars indicate no significant differences. In all groups of data, €? was lower than .01. 


Vaccination Willingness 


Significantly more respondents from big cities (with a population of over 50 thousand residents) were willing to 
be vaccinated compared to respondents from smaller towns and villages (with a population of less than 50 thousand 
residents) (p < .001, p =.118). Moreover, the difference in microbiological awareness in view of willingness to get 
vaccinated was slightly higher in cities (8.7 (SD = 2.7) vs. 8.0 (SD = 2.6)) than in small towns and villages (8.5 (SD = 
2.7) vs. 8.1 (SD = 2.7)). However, these dissimilarities were statistically insignificant (p =.128 and p =.06, respectively). 

No correlation was found between the willingness to get vaccinated and having a medical practitioner among 
relatives (p = .060), the lifestyle (p = .214), studying biology at the extended level (p = .929), nor any other subjects 
at the extended level. 


Reasoning 


The most frequently designated reasons for being against or hesitant towards the vaccination were the fear 
of side effects, insufficient research on the COVID-19 vaccine before its wide-range use, and the belief in the lack of 
vaccines’ efficacy in preventing diseases in general (Figure 4). The will to vaccinate oneself was motivated mainly by 
the expected protection against the severe course of COVID-19 for the respondent, the will to end the pandemic, 
and the protection of relatives and loved ones (Figure 5). 
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Figure 4 


Reasons Why Respondents Do Not Want to Get Vaccinated against COVID-19 
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Discussion 


Among reasons against vaccination against SARS-CoV-2 that the respondents of the questionnaire indicated, 
the most common were: 
e the vaccine has been available on the market for too short a time 
e insufficient information about the vaccine; 
¢ potential post-vaccination reactions and unwanted, long-term side effects (Figure 4, Figure 5). 

In other studies, conducted so far (Al-Mistarehi et al., 2021; Scherer et al., 2021), the same reasons were indi- 
cated by 40-70% of respondents who were undecided or unwilling to get vaccinated. 

The present results also confirmed the hypothesis that the school class profile influences microbiological 
awareness, where the most influential subjects were biology and chemistry (Figure 2). Surprisingly, even though it 
was possible to confirm that deepened knowledge of biology increases microbiological awareness, no clear proof 
was found that microbiological awareness increases the willingness to vaccinate. No relationship was observed 
between learning biology and the willingness to vaccinate either. This is, however, consistent with psychology 
studies implying that a higher level of education or intelligence may not always lead to challenging one’s preexist- 
ing beliefs, but it may instead be used to support them (Driscoll, 2019; Hornsey, 2016; Lewandowsky & Oberauer, 
2016; Sobkowicz & Sobkowicz, 2021). 

Moreover, the obtained results may suggest that the presence of a medical practitioner among relatives 
influences microbiological awareness (Figure 3); however, the €? was very low (see Results). To understand the 
anti-vaccination group's activity, Sobkowicz and Sobkowicz (2021) developed an Agent-Based Model that included 
four agent entities: doctors, patients, initiators, and messages in the infosphere. All agents were given an opinion 
about vaccination ranging from -2 (strongly against vaccines) to +2 (strongly supporting vaccines). The model 
assumed that patients might visit a doctor or get access to official information about vaccines and adjust their 
opinion according to the algorithm described in the publication. However, the doctors’ opinion was prepared in 
a special way as it may have varied from +1 to +2 since some medical practitioners, for many reasons, may not 
want to get engaged in discussions about vaccines with their patients. As shown by the model, radicalizing anti- 
vaccination messages increases the number of unvaccinated patients. The trend may be reversed if the doctor's 
impact is greater than the messaging and if the number of anti-vaccination activists is low. Consequently, it may 
be that medical practitioners’ influence has some potential to indirectly shift public opinion in favor of vaccinations 
by increasing microbiological awareness. However, the study conducted here does not provide clear evidence to 
support this conclusion. 

In addition, it was impossible to confirm the phenomenon observed in the study conducted by Fazel et al. 
(2021). Researchers from Oxford have observed that people who are hesitant towards or against vaccinations are 
more likely to lead an unhealthy life (little exercise, smoking) and spend a lot of time on social media. It is imaginable 
that this relationship is absent among high school students in Poland. Nevertheless, data collected by Fazel et al. 
(2021) are more detailed and assess participants’ lifestyles more accurately. Consequently, the lack of relationship 
was probably caused by the imperfection of the present study. 

Lastly, the difference in the level of microbiological awareness between the vaccinated and unvaccinated 
living in cities was slightly bigger than the difference between the same groups living in smaller towns or villages 
in view of vaccination willingness. However, there was no direct difference in microbiological awareness between 
urban and rural students. 

Therefore, it is plausible that in towns and villages, the social pressure not to vaccinate is more dominant 
than in cities, and when the pressure gets mitigated, microbiological awareness becomes more important. It is 
also worth mentioning that people living in rural areas are more likely to conform due to the lack of anonymity. 
Bourke et al. (2004) mentioned that although urban and rural citizens have approximately the same number of 
strong ties among their respective communities, rural residents usually have a significantly lower number of weak 
ties. This can result in a lack of anonymity among rural citizens compared to urban citizens. 

It is also important to note that the introductory 1999 structural reform aimed to improve the quality of educa- 
tion in the rural areas of Poland (Wisniewski & Zahorska, 2020). It succeeded in reducing the differences between 
urban and rural areas (Jakubowski, 2020), which is also proved by this study, considering that all questions were 
constructed based on Polish curriculum requirements (Table S2) and no differences in microbiological awareness 
between urban and rural areas were observed in high school students. 

As a consequence of the 1999 reform, lower secondary schools prioritized hiring experts in fields such as 


758 


ASS https://doi.org/10.33225/jbse/23.22.749 


Journal of Baltic Science Education, Vol. 22, No. 5, 2023 


i ‘ MICROBIOLOGICAL AWARENESS AMONG UPPER-SECONDARY SCHOOL STUDENTS IN THE 
ISSN 1648 3898 Ne CONTEXT OF COVID-19 VACCINATION 
ISSN 2538-7138 /oniine/ (PP.749-766) 


physics, biology, and chemistry as teachers and financing equipment for laboratory classes (Wisniewski & Zahorska, 
2020). After the 2017 reform, numerous lower secondary schools were not closed down but became a formal part 
of local primary schools. Thus, younger children could continue classes in the same primary school they used to 
attend, and adolescents had lessons in a former lower secondary school building. Therefore, the lack of significant 
difference in microbiological awareness between rural and urban areas could result from the equipment and ex- 
pert teachers that stayed in schools even after the 2017 reform. It may be speculated that the differences in health 
education that were present before the 1999 reform may result in the present social pressure not to vaccinate, 
which older inhabitants in rural areas could create. Consequently, high school students living in towns and villages 
are less willing to vaccinate, which was proved by the present study (see Results). 

Given all of the obtained results, it may be presumed that microbiological awareness is not the most influ- 
ential factor, and other components like filter bubble, social pressure, guardian consent, and cultural background 
may be more crucial in deciding whether or not to vaccinate. For example, it was suggested by Kahan et al. (2017) 
that it is a scientific curiosity rather than IQ or level of education that may lead to contradicting one’s viewpoint. 
Considering the present results, it could be speculated that the level of biological knowledge can indirectly affect 
the willingness to vaccinate but is not the most important factor in the decision process. 

It is important to note that by the time the study was conducted, barely half of children and adolescents in 
Poland (0-17 years old) and 57% of the general public had been vaccinated (Information Service of Republic of 
Poland, 2023; Statistics Poland, 2022), which is similar to the percentage of people declaring their willingness to 
be vaccinated. It can be concluded from this that most people with a favorable attitude toward vaccinations have 
already been vaccinated. Therefore, to increase vaccination coverage in the population, reaching those not in favor 
of vaccination is necessary. The present research shows that it is important to dispel all doubts about vaccinations 
and post-vaccination reactions among young people (Figure 4). 

It could also be beneficial to rerun the study with a bigger group of students and a questionnaire that mea- 
sures scientific curiosity, especially considering that there was no clear evidence for a difference in microbiological 
awareness between the vaccinated and the unvaccinated. Another option worth considering is rerunning the study 
to obtain more data on the social aspects of unwillingness to vaccinate and microbiological awareness in society 
as a whole rather than in high school students. 

Last but not least, adding more questions should be considered to measure microbiological awareness more 
accurately before rerunning such a study. This would considerably diminish the issue with low w hierarchical and w 
total coefficients compared to w asymptotic, which were obtained in the quality analysis of the knowledge test. In 
several cases, some researchers would also regard the corrected item-total correlations as unsatisfactory (Ladhari, 
2010). Reevaluation of tasks is advised if a similar study is to be conducted. 


Conclusions and Implications 


The study confirms that the class profile influences microbiological awareness in a significant way, where the 
most influential subjects were biology and chemistry. Nevertheless, learning those subjects in an extended scope 
did not increase the willingness to vaccinate. It also remained unresolved whether microbiological awareness af- 
fects vaccination willingness. Data collected did not provide clear proof that factors such as lifestyle or a medical 
practitioner among relatives influence the level of microbiological awareness. The abovementioned factors did 
not change the desire to get vaccinated either. 

It was determined that students’ place of residence is an important factor in considering vaccination willing- 
ness. High school students living in small towns and villages in Poland were less willing to be vaccinated. It may 
be due to the higher social pressure of distrusting and opposing vaccinations, which might be more prevalent in 
rural areas than in big cities. Consequently, higher conformity of people living in rural areas could have a negative 
effect on vaccination willingness among high school students from towns and villages. Therefore, more effort 
should be put into health education and vaccination encouragement of society as a whole. 

The reasons for vaccination unwillingness among Polish high school students were unraveled. It is advised 
that biology teachers use these results to address the uncertainties of students considering not only COVID-19 but 
also other illnesses. This could contribute to maximizing vaccinations in the Polish population. 

It is suggested that a study with a bigger group and a methodology to measure scientific curiosity rather than 
microbiological knowledge might be necessary to determine the effect of education on vaccination willingness. 
Analyzing factors such as online infosphere would help resolve the relationship between the lifestyle or relative 
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medical practitioner and the desire to get vaccinated. Larger scale research would also be useful to unravel whether 
the obtained, slight impact of lifestyle and relative medical practitioner on microbiological awareness is truly sig- 
nificant or just a result of the size of the study. 

This preliminary but informative study shows how complex research on the willingness to vaccinate really is. 
Any future studies should focus on measuring a person's education or knowledge and factors like scientific curios- 
ity, social pressure, and online community. 
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Appendices 


Supplementary Table 1 (Table S1) - Table containing questions asked in an online questionnaire with the specified 
aim for each question. 


Supplementary Table 2 (Table $2) - Table containing learning objectives and curriculum requirements on which 
questions were based. 


Supplementary Table 3 (Table S3) - Table containing the exact average score values and standard deviations of scores 
obtained in the test for students attending each subject studied on the extended level and those not attending it. 


Table $1 


Questions Asked in an Online Survey with the Specified Aim for Each Question 


No. 


Question from the survey 


Following questions aim to test students’ knowledge and microbiological awareness. 


1. 


In treatment of which diseases antibiotics are used? 
a. Bacterial diseases 

b. Viral diseases 

c. Fungal diseases 

d. | don’t know 


Aim of the question 


The question tests if participants possess knowledge about 
antibiotics usage in healthcare. 


“All bacteria are harmful” - do you agree with this statement? 
a. Yes 

b. No 

c. | don’t know 


The question tests if participants possess knowledge about 
symbiotic bacteria. 


Finish the sentence (you can choose more than one answer): 
“Antibiotics are effective in the treatment of...” 


Flu 

Quinsy 
Mononucleosis 
Common cold 
Lyme disease 
Gonorrhoea 
Tetanus 
Mumps 
Chickenpox 
j. COVID-19 

k. | don’t know 


za *7e9 a9 5 


The question tests if participants possess knowledge about 
antibiotics usage in disease treatment and pathogenesis of 
highlighted diseases. 


Which microorganisms listed below can cause pneumonia? 
a. Bacteria 

b. Viruses 

c. Fungi 

d. | don’t know 


The question tests if participants know which microorganisms 
can cause pneumonia. 


Which of the agents listed below are efficient in daily hand sanitation? You 
can choose more than one answer. 


a. Alcohol 20%-40% (e.g., vodka, ardent spirits) 
b. Alcohol 60%-70% 

c. Pure alcohol 96% (spirits) 

d. Water with soap 

e. Cold water 

f. Ready-to-use sanitizers from pharmacy 


The question tests participants’ knowledge about sanitizing 
agents. 
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No. Question from the survey Aim of the question 
6. Franek returned home from school. In the kitchen he found a piece of The question tests participants’ basic knowledge about fungi, 
cake, partially covered in green residue. He cut off the green part and ate _ fungal toxins and healthy eating. 
the rest of the cake. In your opinion, was it safe for his health? 
a. Yes 
b. No 
c. Different answer (submitted by participant) 
7. Does the “five seconds rule” work? The question tests participants’ elementary knowledge about 
a. Yes hygiene basics, healthy eating and health-promoting practices. 
b. No “Five seconds rule” concerns food that fell on the floor and the 
: y : common assumption that it is safe to eat if it was picked up 
c. | don’t know this rule 
before five seconds passed. 
8. What are cyanobacteria? The question tests participants’ knowledge about cyanobacteria. 
a. Small plants NOTE: although the English name “cyanobacteria” explains 
b. Water fungi what kind of organisms they are, in Polish there is no such con- 
; . notation (Polish word for cyanobacteria is “sinice”). 
c. Bacteria 
d. Protists 
e. | don’t know 
9. Why is swimming in cyanobacteria-blooming water dangerous? The question tests participants’ knowledge about cyanobacteria, 


a. Cyanobacteria are pathogenic and can cause e.g., infections of urinary bacterial toxins and algal blooms. 


tract 

Cyanobacteria produce dangerous toxins 

Contact with cyanobacteria can cause skin burns and rashes 
This activity is not dangerous 

| don’t know 


Di Oe OS 


The following questions aim to find participants’ reasons and motivations to vaccinate against COVID-19 and reasons for not wanting to do so. 


10. 


Are you vaccinated against COVID-19? 

a. Yes 

b. No, but! want to 

c. No and | still don’t know if | want to 

d. No, | don’t want to 

If you answered you do not want to vaccinate against COVID-19 in the 
previous question, answer why. 

a. |'m afraid of side effects of vaccination 

b. I consider vaccinations ineffective 

c. Because of cultural or religious reasons 

d. Vaccines are produced unethically 

e. | think vaccines contain microchips or other devices 

f. | think vaccines can cause diseases 

g 


. Vaccine against COVID-19 wasn’t subjected to enough number of trials 


and tests 
h. We cannot be sure if it’s safe 
i. Vaccinations are a form of biological weapon 
j. My parents/caregivers did not allow me to vaccinate 
k. Other answer 


The question aims to determine participants’ willingness to vac- 
cinate against COVID-19. 


The question aims to determine reasons why participants are 
not vaccinated. 
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No. Question from the survey Aim of the question 
12. If you answered you are vaccinated or you are, The question aims to determine reasons why participants are 
willing to vaccinate against COVID-19, vaccinated. 
why 
a. | want to be protected against COVID-19 
b. I want to be protected from severe consequences of COVID-19 
c. [want to protect my relatives 
d. | want to protect people who cannot be vaccinated against COVID-19 
or suffer from immunodeficiency 
e. | want to contribute towards ending of pandemic 
f. Because of social pressure 
g. My parents/caregivers told me to vaccinate 
h. | want to travel without restrictions 
i. Other answer 
Table S2 
Learning Objectives and Curriculum Requirements on Which Questions in the Online Questionnaire Were Based 
Subject Learning objectives Curriculum requirements 
Natural |. Knowledge. IV. Me and my body. Student: 
science 4. Acquiring knowledge about systems of the 1. Lists systems of the human body: skeletal, respiratory, digestive, blood 
human body (skeletal, respiratory, digestive, circulation, reproductive, nervous and explains their basic functions; 
blood circulation, reproductive, nervous). 6. Explains basic rules of self-body care and care about the surrounding 
environment. 
Il. Skills and practical usage of knowledge. 
5. Application of rules concerning self- V. Me and my surroundings. Student: 
healthcare, including disease prevention. 2. Explains ways pathogens infect the human body, explains ways of disease 
prevention. 
Biology in V. Knowledge of factors of human health. Il. Diversity of life. 
primary school — Student: 2. Viruses - non-cellular forms of matter. Student: 
1. Analyzes the connection between their 2) explains ways of spreading and prevention of viral diseases (flu, 


actions and maintaining health and recognizes chickenpox, rubella, mumps, measles, AIDS) 

situations demanding consultation with 

physicians. 3. Bacteria - unicellular organisms. Student: 
1) lists bacterial habitats; 


4) explains ways of spreading and prevention of bacterial diseases 
(tuberculosis, Lyme disease, tetanus, salmonellosis); 

5) explains importance of bacteria in natural environment and human body 
4. Protists - organisms of diverse cellular structure. Student: 

4) explains ways of spreading and prevention of diseases caused by protists 
(toxoplasmosis, malaria). 


6. Fungi - heterotrophic organisms. Student: 
5) explains the role of fungi in the natural environment and human life. 


Ill. Human body 

6. Immune system. Student: 

3) compares the mechanism of action of vaccines and antivenom; lists 
medical indications of their use and justifies the necessity of mandatory 
vaccinations. 

12. Reproduction and development. Student: 

6) explains ways of prevention of sexually transmitted diseases. 
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Biology Extended programmes: 

in upper V. Deepening the knowledge of human health 

secondary factors. Student: 

school 1. plans health-promoting activities; 
2. understands the importance of preventive 
screening and recognizes situations demanding 
consultation with physicians; 
5. acknowledges the importance of research in 
disease prevention. 
Primary programmes: 
Il. Deepening the knowledge of human health 
factors. Student: 
1. plans health-promoting activities; 
2. understands the importance of preventive 
screening and recognizes situations demanding 
consultation with physicians; 
4. acknowledges the importance of research in 
disease prevention. 

Table $3 


Exact Mean Score Values (M) and Standard Deviations (SD) of Scores Obtained in the Test for Students Attending Each Subject 


Studied on Extended Level and Those Not Attending It 


Subject 

M 
Biology 9.1 
Chemistry 9.3 
Mathematics (extended biology included) 8.8 
Physics 78 
Geography 74 
History 75 
Polish 77 
Social studies 7.3 
Foreign language 8.5 
Mathematics (extended biology excluded) 77 


https://doi.org/10.33225/jbse/23.22.749 


Participants studying the subject on 
extended level 


Curriculum for high school, extended programmes: 

VI. Bacteria and archaeans. Student: 

5. explains the importance of bacteria in the natural environment and human 
body, including pathogenic bacteria and bacterial diseases (tuberculosis, 
tetanus, Lyme disease, salmonellosis, syphilis, gonorrhea). 


VII. Fungi. Student: 
5. explains ways of infection and prevention of diseases caused by fungi (skin 
mycosis, reproductive organs mycosis, lung mycosis) 


VIII. Protists. Student: 
5. explains ways of spreading and prevention of diseases caused by protists 
(malaria, toxoplasmosis, lambliosis, amoebiasis, trichomoniasis) 


XIl. Viruses, virions, prions. Student: 

6. explains ways of infection and prevention of viral diseases (rabies, AIDS, 
poliomyelitis, HPV-related diseases, flu, measles, chickenpox, rubella, 
mumps, viral hepatitis, some types of cancer). 


Participants not studying the subject on 
extended level 


SD M SD 
2.8 15 24 
2.7 76 24 
2.8 8.3 27 
2.5 8.6 27 
2.5 8.6 27 
2.5 8.7 27 
2.5 8.7 27 
2.3 8.6 27 
27 8.5 27 
2.3 8.6 2.8 
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INTEGRATION OF 
SOCIOSCIENTIFIC APPROACH 
AND DESIGN THINKING: AN 
ENTREPRENEURIAL CREATIVE 
THINKING MODULE FOR STEM 
EDUCATION 


Sufirman Arifin, 
Nyet Moi Siew 


Introduction 


The application of entrepreneurial thinking in teaching and learning 
can produce potential entrepreneurs capable of innovating new ideas or 
products by leveraging existing opportunities. According to Bacigalupo 
et al., (2016), implementing entrepreneurial thinking in students through 
training and experiences in school will result in students who can think like 
entrepreneurs identifying opportunities in the market and exploring suitable 
ways to capitalize on them. A student with an entrepreneurial mindset will 
always be innovative in solving problems (Nadelson et al., 2018). 

Therefore, a learning model that nurtures entrepreneurial thinking 
in teaching and learning at schools needs to be introduced. Based on this 
need, Buang et al. (2009) aimed to understand the entrepreneurial scientist’s 
thinking that led to innovative science-based products. Their qualitative 
findings indicated that respondents integrated entrepreneurial thinking and 
scientific process skills in producing innovative science-based products. As 
a result, they proposed the Science Entrepreneurial Thinking (SET) learning 
model that can be used as a teaching model in the science curriculum at 
the primary and secondary school levels. Based on integrating science and 
entrepreneurship disciplines, the SET learning model consists of five steps: 
observation, new ideas, innovation, creativity, and value. However, the SET 
learning outcomes conclude with students gathering community perspec- 
tives on product ideas through surveys. Subsequently, the analysis of these 
surveys is presented to their peers in the classroom. The students are not 
guided through the steps to introduce their product ideas to the market. 

New ideas or products may not necessarily guarantee their ability to 
penetrate the market, especially in a rapidly changing technological and 
market environment. These products might be introduced too early or too 
different to be accepted in the market (Perry-Smith & Coff, 2011). Fresh ideas 
or products might hold value, yet their true worth is determined by their 
reception in the market (Della Corte & Del Gaudio, 2017). Therefore, in the 
endeavour to innovate ideas or products, creativity can elevate an idea or 
product to have value in the market (Shepherd & Patzelt, 2011). Creativity 
in the entrepreneurial context not only emphasizes extraordinary or unique 
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Abstract. This study was conducted to 

i) ascertain the validity, reliability, and 
feasibility of a module based on the 
socioscientific issue approach and design 
thinking model (SIA-DT), and ii) assess 

its effects on the entrepreneurial creative 
thinking (ECT) in STEM education. The 

first phase of ECT module validation was 
conducted with the assistance of three 
expert evaluators and 32 students. A 
5-point Likert scale questionnaire and 

the ECT test were used to collect data. 

The second phase consists of evaluation 
using a quasi-experimental design with 

a Pre-Post Test of Non-Equivalent Control 
Groups. A total of 64 Form Four students 
were divided into two groups: SIA-DT (n= 
32) and control (n = 32). The ECT module 
has a high validity value and an acceptable 
Cronbach’s alpha reliability of .92. The ECT 
Module’s feasibility was substantiated by 
amean score of 4.71. The results of the 
independent-sample t-test prove that there 
is a significant difference in the post-test for 
students in the SIA-DT group compared to 
the control group in ECT and five constructs 
of ECT. In conclusion, these findings 
demonstrate that the ECT Module is valid, 
reliable, and feasible in STEM education 
and that it is effective in enhancing 
students’ ECT. 

Keywords: design thinking model, 
entrepreneurial creative thinking, module 
development, socioscientific issue 
approach, STEM education 


Sufirman Arifin, Nyet Moi Siew 


University of Malaysia Sabah, Malaysia 


76/7 


Journal of Baltic Science Education, Vol. 22, No. 5, 2023 


INTEGRATION OF SOCIOSCIENTIFIC APPROACH AND DESIGN THINKING: AN ISSN 1648-3898 /Print/ 
ENTREPRENEURIAL CREATIVE THINKING MODULE FOR STEM EDUCATION 
(PP. 767-780) ISSN 2538-7 1 38 /Online/ 


products, but it should also have the capability to penetrate the market and generate profits (Perry-Smith & Coff, 
2011). 

It is clear here that the element of creativity in entrepreneurship can be highlighted as the creative thinking of 
entrepreneurs in creating products and marketing them creatively to generate profits. Therefore, the application of 
entrepreneurial creative thinking should equip students with fundamental skills and knowledge to innovate and 
market ideas or products, subsequently generating income in a dynamic technological and market environment. 

Kennedy and Odell (2014) noted that STEM education applies the process of designing solutions to complex 
contextual problems using current tools and technologies (p. 246). Entrepreneurial creative thinking opens up space 
for students to think more broadly to explore new ideas in STEM-based problems. Owens et al. (2017) stated that 
the socioscientific issue approach helps students to develop and evaluate arguments related to current issues and 
connect them to the interests of society. Therefore, there is a need in fostering entrepreneurial creative thinking 
in STEM education to increase students’ awareness of current issues in the society and encourage them to think 
outside the box in finding solutions to problems through the design process. 

According to Jacobson et al. (2006), a teaching and learning module as a training package helps students 
understand or master skills or learning units effectively. Hence this research was carried out to determine whether 
the ECT teaching and learning module developed as a result of the integration of socio-scientific approach and 
design thinking in STEM education can develop ECT of students. The necessity of this study also aligns with the 
requirements of the Malaysian Education Development Plan 2013-2025, which encourages the incorporation of 
Entrepreneurship Cross-Curricular Elements into teaching and learning (Ministry of Education Malaysia, 2012). 


Theoretical Framework 


The design and development of the ECT teaching and learning module based on the socio-scientific approach 
and design thinking model are executed through the analysis of various elements, such as the socio-scientific ap- 
proach, design thinking model, implementation of STEM education in schools, understanding of students’ learn- 
ing styles in secondary school, and their correlation with theories and learning models. As the produced module 
is focused on Form Four students in secondary school, it applies both Piaget's Cognitive Constructivism Theory 
(Piaget, 1976) and Social Constructivism Theory (Vygotsky, 1978). 

In line with the research subject, which involves Form Four students, the constructivism theory is appro- 
priate for creating an active learning environment. This theory emphasizes engaging in activities that promote 
active knowledge construction (Dogru & Kalender, 2007), enhancing higher-order thinking (Bogar et al., 2012), 
and training students to think critically, analytically, and forward-thinking (Boghossian, 2012). Form four students 
are categorized at the fourth stage, the formal operational stage, in their cognitive development as proposed by 
Piaget’s Cognitive Constructivism Theory. At this stage, Piaget assumed that children’s patterns of reasoning are 
complete, allowing them to use logic, comprehend concrete and abstract concepts, engage in reasoning, and 
solve problems through assumptions. 

This module also emphasizes cooperative learning through group-based product-creation activities. In line 
with Social Constructivism Theory, Vygotsky also highlights the significance of cooperative learning, which is crucial 
in decision-making processes. This includes assigning tasks in pairs or small groups to maintain student motiva- 
tion. This aligns with the implementation of this study, which divides the research subjects into small groups of 5 
to 6 individuals. Vygotsky believed that for effective learning, students need to engage in self-talk in a motivating 
and guiding manner, gradually internalizing the spoken guidance. This way, students can develop and become 
more competent in specific areas. This internalized interaction forms the basis for teaching thinking skills (McLeod 
Saul, 2007). 

Furthermore, Vygotsky's Social Constructivism Theory suggests the active involvement of students, aiding 
them in exploratory learning, providing guidance through explanations, demonstrations, and verbal instructions, 
as well as delivering information and cooperative learning. This theory emphasizes guided exploration by the 
teacher, where the teacher employs questioning methods to prompt students to seek answers to questions (Kubli, 
2005). This principle also aligns with the implementation of this study, where the socio-scientific approach exposes 
students to social issues and encourages them to find scientific solutions. The posed questions act as guided 
explorations for the students. Although students engage in the exploration process, they still receive assistance 
from knowledgeable teachers and peers. 

The researcher also acknowledges the significance of pedagogical principles in the teaching and learning 
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process in secondary schools, supporting the cognitive and social construction of information and knowledge. 
This study employs the Socioscientific Issue Approach (SIA) Model (Sadler et al., 2017) and the Design Thinking 
(DT) Model (Adapted from Hasso Plattner Institute of Design at Stanford University, 2019). 

According to Sadler et al. (2017), the SIA learning model encompasses three phases. In implementing the first 
phase, students discover or are introduced to a specific issue. Students are guided to engage in discussions about 
the stated issue, building their understanding by connecting it with scientific ideas and societal awareness. In the 
second phase, a process of scientific practice takes place, where students reason about the social and scientific 
components related to the issue from the first phase. In this case, students actively seek connections between the 
social issue, scientific knowledge, and scientific practices, making the issue more relevant for resolution (Sadler et 
al., 2017). Students strive to solve the issue based on societal values during the final phase. They are encouraged 
to synthesize ideas and practices and actively express ethical ideas and opinions. 

The Socioscientific Issue Approach model is integrated with the Design Thinking (DT) model in creating this 
module. Plattner (2019) from the Institute of Design at Stanford University presented five stages of the DT learning 
model as a framework for solving real-world problems. These stages are empathize, define, ideate, prototype, and 
test. In the empathize stage, students seek to understand user needs to enable them to set aside their assump- 
tions and gain direct insights into user needs (Chaudhari, 2021). Students gather user problems obtained during 
the empathize stage in the define stage. They then analyse, observe, and synthesize these problems to determine 
the core issues that require solutions. Students also formulate problem statements during this stage. Moving on 
to the third stage of the DT model, ideation, students focus on generating creative ideas and organizing solution 
ideas for the problem statements. Various ideation techniques such as Brainstorm, Brainwrite, and SCAMPER are 
employed. Brainstorming the Worst and Possible Ideas is usually used to stimulate free thinking, helping students 
develop the best solutions for users (Chaudhari, 2021). Subsequently, in the prototype stage, students are required 
to construct prototypes. At this point, students transform their solution ideas into tangible products. In the final 
stage, testing, students test and receive user feedback on the prototypes they have built. Feedback from the public, 
users, experts, and stakeholders enables students to decide if the prototype aligns with the defined problem-solving 
objectives or if an iterative process is necessary. 

The researcher employs a comprehensive integration method (Swartz & Parks, 1994) to combine the socio- 
scientific issue approach with the design thinking model (SIA-DT). In this method, entrepreneurial creative think- 
ing is identified beforehand and taught simultaneously with the subject content. Comprehensive integration is 
taught using five teaching steps: (i) introduction, (ii) active thinking, (iii) thinking about thinking, (iv) reinforcement 
exercises, and (v) application of thinking. In this context, both components (subject content and entrepreneurial 
creative thinking) are interrelated and implemented in each process. 

For the design and development of the Entrepreneurial Creative Thinking teaching and learning module, the 
researcher adopted the ADDIE Model (Branch, 2010), which consists of five phases: analysis, design, development, 
implementation, and evaluation. The ADDIE Model was chosen because it centres on student-centred learning 
activities and goal-oriented instructional design and enables students to engage in meaningful actions and solve 
problems practically. The theoretical framework for the design and development of the Entrepreneurial Creative 
Thinking teaching and learning module is illustrated in Figure 1. 
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Figure 1 


Theoretical Framework in the Design and Development of the Entrepreneurial Creative Thinking Teaching and Learning Module 


Constructivism Theory 
Cognitive Constructivism (Piaget, 1952) 
Social Constructivism (Vygotsky, 1978) 


Socio-scientific Issue Approach 
Model (Sadler et al., 2017) 
Entrepreneurial 
Creative 
Swartz and Parks Integration Thinking (ECT) 
Model 


(Swartz & Parks, 1994) i. Investigation 
il. New ideas 


ili. Design 
iv. Creation 
v. Commercializ 
Design Thinking Model ation 
(Adapted from Hasso Plattner Institute} 
of Design at Stanford University, 
2019) 


Research Aim and Questions 


This study was conducted to design and develop a teaching and learning module that integrates the socio- 
scientific issue approach and design thinking (SIA-DT) to enhance Entrepreneurial Creative Thinking (ECT) in STEM 
Education for Form Four students. There are three research questions that guide this study: 


i. Is the developed ECT module valid, reliable, and feasible for Form Four students? 

ii. Are there significant differences between pretest and posttest scores for the SIA-DT experimental group 
regarding entrepreneurial creative thinking and all constructs of entrepreneurial creative thinking? 

iii. Are there significant differences in entrepreneurial creative thinking and all constructs of entrepreneurial 
creative thinking between Form Four students in the SIA-DT experimental group and the control group? 


Research Methodology 
Design 


This study employs a descriptive and quasi-experimental research design. The descriptive study assesses the 
validity, reliability, and feasibility of the Entrepreneurial Creative Thinking (ECT) teaching and learning module. 
On the other hand, a quasi-experimental design is applied to determine the effects of the ECT module on the 
entrepreneurial creative thinking of Form Four students. The study was conducted for 12 weeks, from November 
2022 to January 2023. 


Sample 


The descriptive study involves 32 Form Four students and three expert assessors. When assessing the reli- 
ability of a newly developed instrument, a sample size of 30 respondents is considered adequate (Chua, 2011). In 
the quasi-experimental study, a total of 64 Form Four students were involved. They were randomly selected from 
two rural secondary schools in Tawau, Sabah, Malaysia. These students were divided into the SIA-DT experimental 
group (n=32) and the control group (n=32). Among them, 28 (44%) were male, and 36 (56%) were female. About 
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23% of the student's parents worked in the government sector, while the remaining 77% worked in the private 
and self-employed sectors. 


Ethical Considerations 


At the initial stage of the study, the researcher obtained consent from the principals, teachers, and students 
involved as research subjects. All research subjects were provided with informed consent letters to seek permis- 
sion from their parents to participate in the research. The consent letter detailed the students’ involvement in the 
research and the parental agreement, indicating their understanding of the research purpose. All respondents 
were informed about their responses’ confidentiality and were assured that they could withdraw from the study 
without any penalty. 


Descriptive Analysis 
Analysis Phase 


The main objective of developing this Entrepreneurial Creative Thinking (ECT) module is to nurture and 
enhance the entrepreneurial creative thinking of Form Four students in STEM Education taught within the Sci- 
ence subject. To achieve this instructional objective, the researchers conducted an analysis of student needs and 
the context. During the needs analysis phase, the researchers conducted semi-structured interviews with three 
Form Four science teachers in Tawau, Sabah, on November 15, 2022. These interviews aimed to understand how 
to enhance ECT within STEM education. Through these interviews, the Form Four science teachers indicated they 
had limited knowledge about implementing STEM in the Science subject. They stated that despite the media 
coverage of STEM, they had not received formal exposure to STEM implementation in their subject. They also ac- 
knowledged not understanding entrepreneurship concepts clearly and were only familiar with entrepreneurship 
through cross-curricular elements within the classroom. They also expressed their lack of understanding regard- 
ing the socio-scientific issue approach and design thinking. One of the interviewed teachers had heard of the 
socio-scientific issue approach but had not yet implemented it. All the teachers shared feedback that they lacked 
exposure to ECT concepts and its teaching process due to the absence of guidance or learning modules related 
to ECT implementation in secondary schools. The interviews also revealed that the teachers had not received any 
training or courses related to ECT. 

In analysing the students and context, the criteria used were adapted from Carlton, Kicklighter, Jonnalagadda, 
and Shoffner’s idea (2000), which focused on students’ prior knowledge about Unit 1.1,“Personal Protective Equip- 
ment,’ and Unit 10.4, “Health Products,’ within two themes of the KSSM Form 4 Science Curriculum: “Safety and 
Health” and “Chemistry in Medicine and Health:’A questionnaire was utilized to gather information about students’ 
proficiency levels in drawing skills, product creation skills, and using digital technology for commercial purposes. 
A total of 30 students were selected to provide feedback for this analysis. The analysis results indicated that all the 
assessed criteria were at a moderate level. 

Therefore, the findings from this interview have provided strong justification to the researcher for designing 
the ECT Module as a guide for secondary school teachers to enhance Entrepreneurial Creative Thinking (ECT) in 
STEM Education among secondary school students. 


Design and Development Phase of ECT Module 


The ECT Module is designed by adapting the socio-scientific issue approach, which involves introducing social 
issues to students, helping them analyse the social and scientific components of the issues, and encouraging them 
to find solutions based on societal values. To assist students in creating products to address the socio-scientific 
issues presented, the design thinking model is utilized as a guide. These models are integrated through the Swartz 
and Parks (1994) framework for integrating thinking. 

This module has six activities, including daily lesson plans, issues, stimuli, implementation procedures, and 
activity scoring rubrics to assist teachers in the implementation process. These activities are built according to 
the latest Standard Curriculum and Assessment Document for Form 4 Science under the KSSM by the Ministry of 
Education Malaysia (MOE). A total of six learning units have been designed under the themes of Safety and Health 
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and Chemistry in Medicine and Health for the Form 4 Science KSSM subject. These units include Fabric Face Cover, 
Face Shield, Face Mask, Paper Soap, Pocket Hand Sanitizer, and Wet Disinfectant Tissue. Each activity is allocated 
135 minutes. However, the suggested time for conducting these activities can be adjusted according to the school’s 
teaching and learning (TL) schedule, as the steps involving product creation and commercialization are often car- 
ried out outside of regular TL hours. 

This module is developed to cultivate Entrepreneurial Creative Thinking (ECT) in the Science TL at secondary 
schools through five steps: (i) investigation (conducting an investigation focusing on socio-scientific issues while 
considering user needs), (ii) generating new ideas (seeking new ideas through socio-scientific reasoning that fulfils 
user needs), (iii) design (formulating new products through sketching), (iv) creation (constructing products based 
on societal values), and (v) commercialization (introducing products to the community through digital technol- 
ogy). These five steps are adapted from the SET model by Buang et al. (2009), extracted based on the integration 
of steps from the DT model and the socio-scientific approach, as illustrated in Table 1. 


Table 1 
Teaching Entrepreneurial Creative Thinking Through the Integration of Socio-scientific Issues and Design Thinking Process Model 


Process of Design Thinking 


Socio-scientific Issues Approach (DT) Model 


Integration of Socio-scientific Issues 


Model (SIA) 
(Sadler et al., 2017) 


Phase 1 Identifying Issue Focus 
Building Understanding of Socio- 
Scientific Issue 


(Hasso Plattner Institute of 
Design at Stanford University, 2019) 


Building Empathy 
Investigating and Understanding User Needs 


Approach (SIA) 
and Design Thinking (DT) Models 


Investigation 
Conducting an investigation focused on socio- 
scientific issues while considering user needs 


Determining Scope 
Synthesizing findings while considering user 
problems and needs 


New Idea 
Searching for new ideas through socioscientific 
reasoning that meets user needs 


Idea Generation 
Creative generation and organization of ideas 


Phase 2 Engagement 

Involving students in acquiring scientific 
knowledge, scientific practices, and 
socio-scientific reasoning practices 


Design 
Formulating new ideas through sketching 


Phase 3 Synthesizing Ideas and Prototyping Create 


Practices Building a prototype Building a product based on societal values 
Resolving issues based on societal Testi are 
esting Commercialization 
values Testing the prototype to gather feedback from Introducing the product to the public through digital 
users technology 


In planning the activities, the researcher identifies socio-scientific issues that trigger ideas at each step of the 
ECT, ultimately leading to product creation. A socio-scientific issue that resonates with students at this time is the 
outbreak of COVID-19. Issues related to the transmission of COVID-19 are highly relevant to be linked with unit 1.1 
on safety equipment and unit 10.4 on health products under the Safety and Health theme and the Chemistry in 
Medicine and Health theme. Through the steps of ECT, students will attempt to solve the problems users face by 
creating personal protective equipment and health products that can help contain the spread of COVID-19 while 
meeting user needs. Some examples of socio-scientific issues provided are as follows: 


Nowadays, users are advised to use a germ-killing liquid containing alcohol levels between 60 to 80 per cent to elimi- 
nate germs on their hands’ surfaces. Similarly, the same percentage is recommended for alcohol-free germ-killing 
liquids that use a chemical compound called quaternary ammonium, specifically benzalkonium chloride. Alcohol 
concentrations exceeding 80 per cent are less suitable due to their quick-drying nature, which can cause the skin to 
become dry and cracked. Among the types of alcohol-based chemical compounds used in germ-killing liquids are 
ethanol (ethyl alcohol),“rubbing alcohol,’ or “isopropyl alcohol:’The application of the liquid is similar to handwashing, 
but the process is quicker because germ-killing liquids dry rapidly. For maximum effect, it is highly recommended to 
thoroughly wash hands to ensure there is no dirt hindering the germ-killing liquid when applied to the skin's surface. 
The question of how we can create a portable hand sanitiser convenient for users arises. 
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Based on the given socio-scientific issue, students are required to create a product that addresses the prob- 
lem using the guidance of the five steps of ECT. In the first stage, students investigate the socio-scientific issue by 
formulating questions to understand the issue, its relationship with science, and its relevance to society. Through 
engagement and discussions with the real-life community concerning the issue, students can set aside their as- 
sumptions and comprehend the community’s needs or the problems that must be resolved. 

In the second stage, students are provided with a foundation to actively engage in imaginative thinking and 
explore new ideas to solve the problem and meet user needs. Through socio-scientific reasoning practices, while 
considering user problems, students strive to present unique and original ideas. 

In the third stage, students are encouraged to synthesize their ideas and design new products through 
sketching. Considering societal values, students creatively arrange their ideas and select a few that they believe 
are most suitable for solving user problems. These chosen ideas are then translated into sketches. In the fourth 
stage, students translate these sketches into actual product designs. The products they create are constructed 
using local raw materials deemed appropriate for the production of the products. 

In the fifth stage, students test the product by obtaining user feedback and considering ways to commercialize 
the product. This stage involves branding the product, pricing strategies, determining distribution channels, and 
creating marketing campaigns through suitable digital market channels in line with the Fourth Industrial Revolution. 
At this stage, students apply basic digital marketing skills through product, price, place, and promotion strategies, 
aligning them with digital technology advancements to generate revenue. 


Assessment Phase 


The assessment phase is carried out to interpret evidence and the quality of the teaching product before 
and after its implementation (Branch, 2009). As suggested by Branch (2009), the assessment is conducted in two 
phases: before and after the implementation of the ECT Module. Cohen and Swedlik (2018) emphasise that a suit- 
able module exhibits high validity and reliability. 

In the first assessment phase, the researcher conducts expert validation of the content of the ECT Module. 
Rubio et al. (2003) recommended involving at least three experts in the field under study to ensure that the domains 
contained in the assessment instrument truly represent the studied area. The module assessment form is provided 
to the experts to enable them to provide clear feedback on pedagogical content (socio-scientific approach, design 
thinking, and STEM education), an overview of activities, the appropriateness of each lesson plan, learning stan- 
dards, the overall sequence of activity units, the integration of the five steps of entrepreneurial creative thinking, 
and written comments to enhance the module. 

The second assessment phase is conducted to assess the reliability and feasibility of the module. To ensure the 
reliability of the ECT Module, a questionnaire set adapted from Ahmad (2022) is provided to the 32 study subjects 
in the SIA-DT group after the intervention using the module is carried out. Aung et al. (2021) have indicated that 
questionnaires based on activities exhibit higher reliability indices compared to module objectives. The ECT Mod- 
ule comprises six activities, and each activity is assessed by five items using a Likert scale ranging from 1 (strongly 
disagree) to 5 (strongly agree). The items address whether each activity conducted can help respondents cultivate 
the five constructs of entrepreneurial creative thinking following the module's learning outcomes. All 30 items are 
presented in a language that is appropriate and clearly understood. 

To ensure the ECT Module’s implementation in the TL for Form Four students, the researcher distributed a 
questionnaire to 20 Form Four science teachers. The teachers were requested to assess the feasibility level of the 
module by indicating their level of agreement on a scale from one to five for various aspects of the module, in- 
cluding the activities within the module, the integration of SIA and DT, and an overall assessment of the module. 


Quasi-Experimental Study 


The experimental research phase involves the implementation of the ECT teaching and learning module and 
assessing its effectiveness during classroom teaching and learning sessions. In line with this, the quasi-experimental 
Non-Equivalent Control Group Pre-Post Test design is used. A total of 64 Form Four students were randomly selected 
from two rural secondary schools in the Tawau district and divided into the experimental group (SIA-DT, n=32) and 
the control group (n=32). The research subjects in the experimental group received the ECT module, which covers 
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six learning units over 12 weeks from November 2022 to January 2023. Each learning unit in this module consists 
of three sessions that take 180 minutes. A total of 6 teaching periods per week (180 minutes) were allocated for 
Form Four science lessons, each lasting 30 minutes. The researcher suggests that teachers complete one set of 
learning units in the ECT Module for two weeks. One week is dedicated to implementing the steps in the module, 
while the second week involves creating and producing marketing advertisements through digital technology. 
Teachers can also conduct module activities outside the formal teaching periods, depending on availability. On 
the other hand, students in the control group are exposed to the conventional design model production method 
without incorporating the socio-scientific issue approach and design thinking model. 

To assess the module's effectiveness, the Entrepreneurial Creative Thinking Test (ECTT) instrument developed 
by the researcher was utilized. This instrument has been validated, proven reliable, and deemed suitable for evaluat- 
ing the entrepreneurial creative thinking of Form Four students. The ECTT consists of 10 items that require students 
to respond to statements, generate idea sketches, and propose product marketing strategies using technological 
elements. The items were developed based on the five constructs of Entrepreneurial Creative Thinking. The ECTT 
prompts students to address the issue of using face shields and masks separately and in combination. The context 
of producing a combination of face shields and masks was chosen as it is included in the Curriculum and Assess- 
ment Document under the Safety and Health theme. Students were provided with ten-item questions organized 
according to the five ECT constructs—investigation, new idea generation, design, creation, and commercializa- 
tion—to enable them to structure their responses and consequently lead to the desired outcomes. Scoring for 
the ECTT was conducted based on the scoring scheme by Ho et al. (2013). Each item presented in this test carried 
a minimum score of 0 and a maximum score of three. Meanwhile, the scores for each construct ranged from 0 to 
6. All items accumulated a total score of 30. 


Data Analysis 


Descriptive study data was obtained by calculating percentages, means, and standard deviations. On the 
other hand, inferential study data was obtained through analysis using IBM SPSS (version 26). For this analysis, the 
significance level was set at .05. 


Research Results 
Content Validity 


The researcher utilized the services of three expert individuals in entrepreneurial thinking, design thinking, 
and STEM education. In determining the experts’ agreement on content validity, the researcher also employed the 
Content Validity Index (CVI) based on each item on the scale (I-CVI) as well as the overall scale (S-CVI/Ave), following 
the approach outlined by Polit et al. (2017). The obtained Content Validity Index Item (I-CVI) values for the standard 
assessment and learning outcomes aspects were below .78. This suggests that the standard assessment and learn- 
ing outcomes aspects need refinement based on expert group comments and suggestions. The Content Validity 
Index Scale (S-CVI/Ave) value of .93 for the entire module meets the criteria set for new instruments, indicating that 
the content validity of the ECT Module, according to the expert panel, is high and acceptable (Polit et al., 2017). 

In line with the experts’ recommendations, several improvements have been made to the content and ac- 
tivities of the ECT Module; 1) The activities should incorporate an introduction to current issues to help students 
understand the relevance of the topics under discussion, 2) The term‘marketing’ needs to be reviewed and revised, 
3) The alignment of learning standards with learning outcomes for Units 1-3 should be ensured, 4) Clear explana- 
tions regarding product marketing activities should be included, and 5) Illustrations and supplementary materials 
need to be added to each unit in accordance with the demands of core values, similar to those found in textbooks. 


Reliability 
The module's reliability can be determined when study participants can master the objectives and effectively 
follow the steps for each activity in the module at a satisfactory level. Most researchers suggest that an alpha 


coefficient greater than .8 indicates high reliability (Bogden & Milken, 2003; Sekaran & Bougie, 2010). Preliminary 
research results indicate that the Cronbach's Alpha values for all units within the module range from .74 to .88, 
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with an overall Cronbach's Alpha value for the ECT module of .82. This demonstrates that the internal consistency 
of the developed ECT module as a whole is high. 


Table 2 
The Cronbach's Alpha value of the ECT Module 


Unit Unit within the module Cronbach’s Alpha 

1 Face mask fabric 88 
2 Former face mask 87 
3 Face mask 84 
4 Hand wash soap AT 
5 Hand sanitizer 86 
6 Disinfectant wet wipes 14 

Average 83 

Feasibility 


To ensure this ECT Module's implementation in the TL for Form Four students, the researcher distributed 
a questionnaire to 20 Form Four science teachers. The teachers were asked to assess the feasibility level of the 
module by indicating their agreement level on a scale from one to five for aspects related to module activities, 
integration of SIA and DT, and overall assessment of the module. Table 3 presents the teachers’ assessment of the 
feasibility of the module. 


Table 3 
Teacher Assessment of the Feasibility of the ECT Module 


Unit Feasibility Criteria M 
1 Activity implementation 4.65 
2 Integration of SIA and DT 4.76 
3 Overall module 471 

Average 4.71 


Junus et al. (2021) stated that criteria with a minimum threshold of 3.50 indicate that the feasibility aspect 
of the module is acceptable. However, if the minimum threshold is below 3.50, changes need to be made as the 
feasibility aspect of the module is considered unsatisfactory. Overall, teachers provided assessments ranging from 
4.65 to 4.76 for all three criteria related to the feasibility of the module. A minimum of 4.71 indicates that teachers 
agree that the ECT Module can be implemented in schools to foster ECT among Form Four students. 


Effectiveness of the ECT Module 


The Entrepreneurial Creative Thinking (ECT) Module was tested on 32 students in a rural secondary school in 
Tawau, Sabah. Students were given the Entrepreneurial Creative Thinking Test (ECTT) to assess their entrepreneurial 
creative thinking before and after the intervention using the ECT module. Inferential data analysis was conducted 
using paired samples t-test to determine whether there were significant differences between pre-test and post- 
test scores for the study subjects who underwent the SIA-DT intervention. Table 4 presents the results of the 
paired samples t-test analysis for the SIA-DT experimental group. The analysis indicates significant differences for 
pre-test and post-test scores in entrepreneurial creative thinking (t (31) =-20.605, p < .05), Investigation construct 
(t (31) =-12.175, p < .05), New Idea construct (t (31) = -13.373, p < .05), Design construct (t (31) = -13.638, p < .05), 
Create construct (t (31) = -15.033, p < .05), and Commercial construct (t (31) = -12.938, p < .05). 
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Construct Test M (SD) MD (SD) t df p 
Pre 11.59 (3.221) ; 
ECT Hee 20.605 31 <.05 
Post 25.75 (2.185) (3.886) 
Pre 2.47 (1.077) ; 
Investigation re -12.175 31 <.05 
Post 5.13 (.707) (1.234) 
Pre 2.16 (1.139) ; 
New Idea oe 13.373 31 <.05 
Post 5.19 (.738) (1.282) 
Pre 2.13 (1.100) d 
Design ae ( 13,638 31 <.05 
Post 5.13 (.707) 244) 
Pre 2.25 (.916) : 
Create na 15.033 31 <.05 
Post 5.19 (.738) (1.105) 
Pre 2.59 (.875) ae 
Commercial 1 407 -12.938 31 <.05 
Post 5.13 (.707) (1.107) 


Note: Significance level at p= .05 


Table 5, on the other hand, presents the results of the Independent Samples t-Test analysis between the 
experimental group and the control group. The Levene’s test for equality of variances was not significant for ECT 
(p = .236, p > .05), Investigation construct (p =.171, p > .05), New Idea construct (p = .337, p > .05), Design construct 
(p =.129, p > .05), Create construct (p = .420, p > .05), and Commercial construct (p = .109, p > .05), indicating that 
both the experimental and control groups have equal variances. 

The analysis results indicate that the study subjects in the SIA-DT experimental group have significantly 
higher post-test scores compared to the control group in terms of the Entrepreneurial Creative Thinking (ECT) 
aspect (t(62) = 21.962, p < .05), Investigation construct (t(62) = 9.390, p < .05), New Idea construct (t(62) = 13.904, 
p < .05), Design construct (t(62) = 11.988, p < .05), Create construct (t(62) = 16.812, p < .05), and Commercial con- 
struct (t(62) = 6.696, p < .05). These results demonstrate that there is a significant difference in post-test scores for 
students in the SIA-DT group compared to the control group in terms of entrepreneurial creative thinking and all 
five constructs of entrepreneurial creative thinking. 


Table 5 


Results of the Independent Samples t-Test for the experimental Group and the Control Group 


Experimental Group 


Control Group 


Construct M M a t df p 
(SD) (SD) 
ECT ee Ae fen 962 <5 
Investigation ae hee en 9.390 62 <.05 
New Idea he an on 13.904 62 <.05 
Design en aay oh 11.988 62 <.05 
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Experimental Group Control Group MD 
Construct M M (SD) t df p 
(SD) (SD) 
5.19 2.09 3.094 
Create (738) (734) (184) 16.812 62 <.05 
‘ 5.13 4.03 1.094 
Commercial (707) (595) (163) 6.696 62 <.05 
Discussion 


This study aimed to assess the developed module for enhancing entrepreneurial creative thinking among 
Form Four students. The development of the ECT module was geared towards meeting the needs of students in 
the face of the Fourth Industrial Revolution. Assessments of its validity, reliability, feasibility, and effectiveness 
were analysed to ensure its feasibility in schools. The ECT module was constructed based on a clear and detailed 
theoretical framework, incorporating the Socio-scientific Issues Approach and the Design Thinking Model by 
integrating the Swartz and Parks Integration Model (1994) to examine the effects on the five constructs of Entre- 
preneurial Creative Thinking. 

Overall, the study findings have demonstrated that the ECT module possesses good content validity and 
is feasible for enhancing entrepreneurial creative thinking among Form Four students. Analysis regarding the 
content validity of the module has shown that the three expert assessors well-received the ECT Module, with 
several improvements made to various aspects within the module. Reliability analysis indicates that the ECT 
module falls within an acceptable range based on Cronbach's alpha values. This study has confirmed that the 
integration of the Socio-scientific Issues Approach, Design Thinking Model, Swartz and Parks Integration Model 
(1994), entrepreneurial creative thinking model, and the ADDIE instructional design model verifies the legitimacy 
and reliability of the ECT Module for implementation in STEM-based pedagogical practices. 

The results of the paired-sample t-test indicate that Form Four students in both the SIA-DT and control 
groups exhibited better performance in the five constructs of entrepreneurial science thinking in the post-test 
compared to the pre-test. However, Form Four students in the SIA-DT group demonstrated a significantly larger 
improvement in scores compared to their peers in the control learning group. This signifies that the opportunity 
to learn with the ECT teaching and learning module profoundly impacts the entrepreneurial creative thinking 
of Form Four students. As for the independent samples, t-test results suggest that using the ECT teaching and 
learning module effectively nurtures ECT and the five constructs of ECT. 

Exposure to socio-scientific issues in the ECT module enables students to connect these issues with their 
personal experiences. Engaging in arguments about socio-scientific issues alongside inquiry encourages students 
to form preliminary ideas about the conceptualization of the product to be created, provides a clear understand- 
ing of the subsequent steps to take, and assists students in easily seeking answers to the issues or problems they 
encounter (Mutvei et al., 2017). Schmidt et al. (2013) emphasize that effective inquiry focuses on problem-solving 
rather than problem-finding. This is consistent with Fillis and Rentschler (2010), who assert that entrepreneurial 
creative thinking should prioritize the continuous creation of alternative solutions to address problems and 
identify new opportunities. 

Furthermore, exposure to socio-scientific issues through the ECT Module aids in developing skills such 
as decision-making, evaluating statements, analysing evidence, and assessing various perspectives through 
discussions and debates (Sadler & Zeidler, 2005; Zeidler, 2016; Zeidler & Nichols, 2009) within the context of the 
real world. In this regard, students nurtured through SIA will be more adept at analysing and solving problems 
precisely. Moreover, the scientific knowledge generated through debating socio-scientific issues encourages 
students to generate ideas more effectively towards finding solutions to societal problems, thereby enhancing 
entrepreneurial thinking among students (Kinslow et al., 2017). Aligned with the socio-scientific issues addressed 
in each activity unit, these issues allow students to analyse and provide input, enabling them to create product 
innovations that address the issues raised. 

Additionally, the increase in minimum scores among students participating in SIA-DT compared to control 
group can be elucidated through the highly detailed approach to socio-scientific issues (Topcu et al., 2018). In the 
initial phase of the socio-scientific issues approach, which engages students in issue-focused activities, students 
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investigate the matters discussed within the issues. Subsequently, in the second phase involving connecting 
scientific ideas and relating to societal sensitivities, students pose questions among group members, link experi- 
ences to discoveries, acquire new knowledge, and establish connections between events or objects. The utiliza- 
tion of the socio-scientific issues approach, supported by the design thinking model, aids in enhancing students’ 
entrepreneurial creative thinking levels. This is because the discussions surrounding socio-scientific issues lead 
to a step-by-step product generation process, as exposed by the design thinking model (Mutvei et al., 2019). 

The integration of the socio-scientific issues approach with the design thinking model also encourages 
students to think outside the box and generate innovative ideas when seeking critical problem solutions (Aung 
et al., 2021). Therefore, design thinking can address ambiguous problems (Aung et al., 2021; Buchanan, 2001). 
Such problems can be effectively solved by employing high-level thinking skills. According to the revised Bloom's 
Taxonomy educational objectives (Anderson et al., 2001), design thinking encompasses all cognitive activities, 
including remembering, understanding, applying, analysing, evaluating, and creating (Powell & Kalina, 2009). 

The degree of improvement in students’ entrepreneurial creative thinking levels across all five constructs 
using the SIA-DT method as opposed to the conventional method is facilitated by the systematic integration of 
SIA and DT. Kinslow and Sadler (2018) and Sadler et al. (2017) also support that combining SIA with other in- 
structional models enhances students’ scientific knowledge as they explore socio-scientific issues. The debate of 
socio-scientific issues leads to discussions of ideas that expand scientific knowledge and its implications within 
society, ensuring deep understanding to make well-informed decisions. Consequently, SIA guides the appropri- 
ate decisions when selecting the best new ideas for further development into student-created products (Sadler 
& Zeidler, 2005; Zeidler, 2016; Zeidler & Nichols, 2009). 

Indeed, implementing the ECT aims to cultivate creative entrepreneurs prepared to face the challenges 
they will encounter when designing and launching new ventures. Therefore, integrating the ECT into students’ 
education through training and experiences in schools will produce students who can think entrepreneurially, 
identify market opportunities, and explore suitable ways to market the product (Bacigalupo et al., 2016). Stu- 
dents equipped with entrepreneurial creative thinking will consistently exhibit innovation in problem-solving 
(Nadelson et al., 2018). In other words, integrating the ECT into learning activities and instructional facilitation 
within the classroom can foster individuals capable of innovating and commercialising new ideas or products 
by leveraging existing opportunities. 


Conclusions and Implications 


This study confirms that the developed ECT teaching and learning module exhibits acceptable reliability and 
content validity. Overall, the ECT module is feasible to foster the entrepreneurial creative thinking among Form 
Four students. The study shows that the integration of the socio-scientific issues approach and design thinking 
model in developing the ECT module has effectively increased student entrepreneurial creative thinking. This 
study demonstrates that a carefully designed integrated approach for teaching ECT enables students to investigate 
the socio-scientific issues, generate new ideas, design, create and commercialize products in order to address the 
raised issues with the assistance of systematic steps within the design thinking model. 

The present research findings draw the two implications for current STEM education practices. First, the 
existing STEM education usually adopts a particular teaching approach or model which cannot help promote 
the diversity and comprehensiveness of the STEM education to foster student's entrepreneurial creative thinking. 
Teachers should actively explore new integrated teaching approach to make the curriculum more meaningful 
to students. Second, teachers need continuous professional development to improve their ability to utilize new 
integrated teaching approach. Educational administrations should provide teachers with training and resource 
support and encourage them to innovate their teaching approach. At the same time, teachers need to upgrade 
themselves to better meet the demands of 21st Century education. 

The future development of teaching and learning module requires the joint efforts of teachers, schools, and 
administrations to create a supportive atmosphere for innovations in integrated teaching approach to effectively 
stimulate students’ entrepreneurial creative thinking in STEM education. 
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Limitations and Future Research 


This study has two limitations that provide avenues for future research in the assessment of the ECT Module. 
First, this study was a one-time study in the specific context of STEM education for Form Four students in two sec- 
ondary schools. Therefore, the findings may not be applicable to different age groups. Second, the data analysis was 
based on the interpretation of the statistical results which lacked in-depth narrative evidence such as interviews 
to support the triangulation and coding results of classroom observations. As a result, future researchers are rec- 
ommended to replicate the study in more diverse contexts; and to focus on a variety of empirical data, including 
observations, interviews, and document analyses. 
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THE EFFECT OF OUTDOOR 
INQUIRY PROGRAM FOR 
LEARNING BIOLOGY USING 
DIGITAL TWIN TECHNOLOGY 


Jung-ho Byeon, 
Yong-Ju Kwon 


Introduction 


In the 21st century, rapid digitalization has progressed throughout soci- 
ety, but changes in education have been slower than in other fields. In science 
education, there have been attempts to apply various digital technologies for 
effective learning from a convergence perspective, such as using the virtual 
world and artificial intelligence. Metaverse technology such as virtual and 
augmented reality have been applied to dangerous experiments, repetitive 
attempts or pilot tests, ecosystems in cyberspace, and museum experience 
programs. However, due to one-time activities that aroused interest, difficul- 
ties in applying to each school class due to technical barriers to entry, and 
negative perceptions of digitalization, it did not lead to significant changes 
or an overall paradigm shift. 

On the other hand, the sudden COVID-19 pandemic has acted as a factor 
accelerating the digitization of the social system, including education, and 
has called for a radical paradigm shift in education (Fauville et al., 2021). It 
also served as an opportunity to reveal the digital infrastructure gap held 
by schools, the digital teaching and learning competency gap of teachers, 
and the digital literacy gap of students (Reimers, 2022). On the other hand, 
the change in the educational paradigm using Edutech enables time-space 
expansion of learning by continuously and actively utilizing advanced tech- 
nology such as artificial intelligence and virtual reality in learning activities 
beyond the level of experience for simple interest (Holmes et al., 2019). 

Post-pandemic education requires different learning methods in offline 
and online contexts, and expanding educational activities using virtual space 
presents new possibilities for science teaching and learning. In particular, 
learning biology has the new possibility of teaching and learning to utilize the 
virtual world online compared to other scientific education areas. Learning 
biology has successful learning effects when indoor and outdoor activities 
are carried out in parallel or together, and activities to observe and explore 
living organisms in a real natural environment are known to have various 
effects on learners’ affective areas (Wells et al., 2015). It is generally agreed 
that providing outdoor inquiry activities is essential for learning biology, 
but the actual performing of outdoor activities is complex due to practi- 
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cal limitations (Behrendt & Franklin, 2014; Dillon et al., 2006). In order to observe and explore various organisms 
outdoors, one needs help not only from experts or biological guides but also students must be able to use illustra- 
tions or auxiliary materials to expect the effect of inquiry learning outdoors. The physical collection has recently 
been limited for normative reasons and bioethics, and visiting time is limited due to the nature of the outdoors 
(Behrendt & Franklin, 2014). 

Alternative learning activities using metaverse, such as indirect experience videos and virtual experience 
activities, are provided to compensate for this problem, but simply presenting digitized three-dimensional spaces 
has limitations in conveying diversity and reality in the real world. In addition, the high level of digital literacy and 
professional equipment required to create virtual spaces is another reason for the low utilization of metaverse tech- 
nology in the educational field. Ultimately, the digital gap between students, teachers, and students can also affect 
bio-learning activities using the virtual world. Therefore, teachers should try to provide an alternative environment 
where students can freely continue to explore and learn about living organisms by overcoming the limitations of 
outdoor inquiry activities. Then, what is some alternative learning method to overcome various limitations like 
the digital divide in education using metaverse, and what teaching and learning strategies can enhance students’ 
interest in learning biology and learning sustainability? 


Digital Twin as an Alternative for Learning Biology Outdoor and Indoor 


Recently, learning using the metaverse platform has been proposed as an alternative activity that can com- 
pensate for providing an interactive experience similar to reality (Dwivedi et al., 2022; Kuznetcova et al., 2019). 
Metaverse is generally used as a term for an environment in which one can interact and communicate with others 
ina virtual space. Generally, the metaverse can divide into four types; virtual reality (VR) creates a three-dimension 
space similar to reality, augmented reality (AR) can augment and possibly brings digital content to the real world, 
life-logging records and shares real-world data in digital space, and mirror-world moves the real world to virtual 
space (Tilak et al., 2020). 

Among the types of the metaverse, virtual reality is a digital implementation of a similar space to the real world 
and has the advantage of providing a sense of immersion by interacting with content. In particular, it can provide 
a sense of space and freedom, so teachers or students can easily visit and experience places with spatiotemporal 
constraints in cyberspace. On the other hand, it is difficult for anyone to implement it quickly because virtual real- 
ity requires 3D rendering technology, and it is only possible to add or modify various contents if they are experts 
(Kelton, 2007; Suzuki et al., 2020). 

The virtual world can provide a sense of immersion as a space that imitates the real world, but despite vari- 
ous advantages, the virtual world generated by prior VR technology does not reflect the real-world information 
equally, like length, width, and size (Tao et al., 2019). However, digital twin technology that implements real- 
world information equally in the virtual world is providing a solution to these deposits of virtual reality due to 
recent improvements in image processing hardware and artificial intelligence technology (Liu et al., 2021; Tao 
& Zhang, 2017). 

The digital twin is a virtual replica that generates the same output value as a physical object for an input 
value and can define as an algorithm that describes a physical object's potential or actual components (Grieves 
& Vickers, 2017; S6derberg et al., 2017). The digital twin concept includes the components of an existing physical 
object, a replica of a virtual world, the transfer of data from a physical object to a virtual replica, and the flow of 
data from a virtual replica to a physical object (Thelen et al., 2022). The digital twin was proposed 20 years ago but 
was first implemented in 2011 to reproduce aircraft rescue behavior digitally (Tuegel et al., 2011), and has since 
been recognized as a method for product management and production in various fields and is now recognized as 
a significant high-tech technology leading the future industry (Lu et al., 2020). 

The development of big data and artificial intelligence expands the range of digital twins and enables the 
construction of twins by digitally replicating physical entities, processes, and systems, such as living and inanimate 
objects. In addition, real-time monitoring is possible by improving the interconnection between real-world physi- 
cal objects to share data with the goal of optimization and digital replicas in cyberspace (Rathore et al., 2021). The 
digital twin is a technology that enables innovative work, monitoring, diagnosis, and prediction in various fields, 
and its utilization in education is very high. In particular, in terms of mediating online learning and authentic 
learning, digital twins have the advantage of expanding opportunities for practice or hands-on learning to infinity 
(Rathore et al., 2021; Sepasgozar, 2020). 
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Therefore, utilizing digital twin technology can overcome the limitations of one-time activities, passive 
experiences, and a lack of realism in outdoor learning biology using the metaverse. Furthermore, if the spaces 
and characteristics of living organisms in outdoor inquiry activities are replicated in the virtual world, continuous 
learning and monitoring can also be achievable indoors. 


The Strategy of Metaverse Utilization for Learning Biology Outdoor and Indoor 


In creative thinking, learning is effective when experiencing indoor learning activities for knowledge genera- 
tion and understanding concepts and collaborative inquiry activities for practical experience (Pulgar, 2021; van 
der Zanden et al., 2020). Especially in the case of learning about living things, the learning effect is excellent when 
indoor and outdoor activities are combined, and activities of observation and exploration of living organisms in 
the natural environment are known to provide a positive effect on the learner's affective domain (Wells et al., 2015). 
Therefore, indoor and outdoor activities for learners should be necessary for the practical application of learning 
biology. However, some restrictions exist, like the lack of teachers’ expertise in outdoor activities, the difficulty of 
on-site visiting, and the activity time constraints (Behrendt & Franklin, 2014; Dillon et al., 2006). New methods using 
virtual experience were suggested as alternative activities to overcome these problems shown during field trips. 
However, evaluating the use in the school field is inconsistent regarding continuity and effectiveness (Behrendt 
& Franklin, 2014). 

After the COVID-19 pandemic, the metaverse expanded to everyday use in various fields, and the possibility 
of educational usage is newly emerging (Kuznetcova et al., 2019). These days, if the biology teacher can effectively 
apply the interactive learning method in the virtual world to the real world, they could exceed the spatial limit of 
learning activity. For example, the learning effect could improve on the class having spatial difficulty due to the 
limited activity places, such as exploring living organisms in the field. 

Considering the utilization of the metaverse in the educational field, biology teachers should be able to build 
a virtual world quickly, and students should be able to operate various contents and functions of the metaverse. 
Also, teachers can save time and expand the activity space required for biology inquiry and learning through the 
metaverse. If teachers want to investigate and classify organisms in a convergent method using the metaverse, 
they could go outdoors with students to explore and collect them after the hands-on learning activity. For the 
construction of the virtual world, they should overcome various limitations, such as the selection of the inquiry 
site, the method of transportation, the presence or absence of target organisms, a preliminary survey of the visit- 
ing route on-site, damage to the ecosystem due to organism collection, and weather on D-day. Also, students and 
teachers in the virtual world could conduct social interactions between users and content. 

Therefore, this new interaction method affects the learner's affective domain, like motivation, attitude, and task 
commitment (Akour et al., 2022; Stokel-Walker, 2022). Therefore, it is necessary to organize and utilize the virtual 
world for places teachers and students can quickly and frequently visit to combine learning biology indoors and 
outdoors easily. Also, it needs to consider the effect on students’ affective domain in terms of teaching and learning 
and organize classes using the virtual world according to the school’s situation and the student's characteristics. 


Research Aim 


It is essential to develop teaching and learning methods to guide the effective use of digital technology to 
respond appropriately to changes in the educational field, such as the increased possibility of activities in cyber- 
space due to the expansion of digital technology in education (Petrie, 2022; Vincent-Lancrin, 2022). Also, class 
activities with various digital technologies can provide customized learning opportunities in learners’ problem- 
solving process experience by expanding learners’ information search and utilization (Collins & Halverson, 2018; 
Yair et al., 2001). In this context, it has made challenging attempts to develop learning programs using advanced 
digital technologies such as artificial intelligence and the metaverse in various teaching areas. Also, recently with 
the development of Al technology, in the case of metaverse platforms, entry barriers to using innovative technolo- 
gies using digital twin technology, a concept of replicating real-world spaces and objects to virtual spaces, are 
decreasing (Liu et al., 2021; Rathore et al., 2021). 

If anyone can easily replicate real space in the virtual world with the development of artificial intelligence 
technology, teachers and students who do not have professional digital knowledge can implement the virtual 
world and effectively use it in classes. Then, how can anyone easily use the virtual world in their classes without 
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professional knowledge as an alternative activity to overcome the limitations of outdoor activities in biology classes? 
Therefore, the researchers established the following research aims to address these questions. 

First, it was intended to create a virtual world with a high sense of reality by using the digital twin and to 
organize a class program that allows students to learn biology. Students should be able to easily create the virtual 
world without using complex digital devices and technologies to solve the problem of high entry barriers in us- 
ing the virtual world. In addition, to resolve the lack of reality, it is necessary to replicate the size of the space, the 
characteristics and size of the object, and the shape of the space. Therefore, researchers set the aim to create a 
virtual replica using a digital twin platform by simple digital devices in the school field and organize a class program 
that explores living things. 

Second, it was organized into an experimental group and a control group to confirm the main effect and group 
effect to confirm the impact of the experience of digital twin classes on the student's affective area. Exploration of 
living things outdoors can affect the learning motivation and tendency to continue learning. However, the positive 
impact on students’ affective areas is somewhat low or case-by-case due to low learner initiative and difficulty in 
solving professional problems. In addition, experience in learning biology using the virtual world also has a lower 
noticeable effect on the student's affective domain because of a low sense of reality and lack of interaction. There- 
fore, researchers set the goal to analyze the main effect of the experience of learning biology programs indoors 
and outdoors on the student's affective domain and the group effect of using digital twin technology. 


Research Methodology 
General Background 


It can induce immersion in learning motivation and activities to students when learners are free to organize, 
control, and control objects in science experience activities using the virtual world (Dede et al., 2017; Psotka, 1995). 
In this context, it was intended to build a virtual world using a metaverse platform with digital twin technology 
replicating and manipulating the real world’s space in cyberspace. In addition, students can freely explore any 
time and place when they can easily access living organisms outdoors (Subramaniam et al., 2018), and foster a 
positive attitude toward students’ biological and ecological environment when outdoor experience activities are 
easy to access (Farmer et al., 2007). Therefore, the garden plant in the school was selected as a digital twin target 
as a place where data on the space corresponding to the physical object that is the target of the digital twin can be 
continuously obtained. This study set up research aims to clarify the utilization possibility and the effect of digital 
twin utilization learning programs on students’ affective domain in biology class. Most of all, students must have 
the essential skill of digital literacy for generating and operating the digital twin, and they need to be taking the 
biology curriculum to inquire about living organisms. In addition, since it is necessary for anyone to easily create 
and operate a digital twin, high school students who freely handle digital devices such as smartphones and have 
basic knowledge of biology were selected as the subjects of the study. 


Participants 


From the perspective of creating digital twins, students need a certain level of digital device knowledge 
to acquire image data for virtual space construction self-directedly, adjust the location of virtual space in detail 
compared to physical space, and upload interactive content. In addition, basic knowledge of identification and 
classification is required for students to produce bio-related content to be presented in a virtual space. According 
to these needs, high school students in Korea aged 18 or older were recruited per this study's digital twin software 
policy. While most schools disagree with consistent classes using the digital twin, forty-four students studying at 
school A in South Korea participated in this study. The researcher received the school’s official permission and stu- 
dents’ voluntary research participation. Therefore, two classes in that school were recruited as application places 
to test research aims. 

Forty-four students from a public school in Korea participated in class activities to replicate school garden 
spaces and organisms in digital spaces and to check the effect of learning biology programs on students’ affec- 
tive areas. All the students were taking ‘Biology II’ courses and completed the ‘Biology I’ course, which included 
a national curriculum on biology in Korea. Students were divided into different classes to ensure interaction and 
team activities in students’ activities, and the students were divided into a plant classification program group and 
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a plant classification class group using digital twin technology to check changes before and after the learning 
program experience. 

The purpose and contents of the study were explained to all participating students before conducting the 
class program and pre-examination, and students submitted a voluntary participation agreement. The response 
to pre-test results was obtained by providing a survey paper to confirm changes in the affective domain before 
starting the first activity, and students’ follow-up responses were obtained using the same survey after the last class 
activity. Among the students who responded, three were omitted from the analysis due to partially participating in 
the class for absence or some test questions. Therefore, the results of the pre-post response of twenty-one people 
in the control group who conducted plant classification classes and twenty people in the experimental group who 
conducted plant classification classes using digital twins were used for analysis. 

An introductory survey was conducted during the pre-test period to confirm the individual metaverse experi- 
ences of the students who participated in the study (Table 1). All the students had experienced a metaverse, which 
was mainly divided into the first experiences in elementary or high school. 68.29% of students were experiencing 
metaverse in game type, while 31.71% of students were experiencing learning or other types. In response to the 
preference for metaverse, 73.17% said good, and no student said dislike. In addition, 56.1% of students frequently 
contacted a metaverse, and only 2.44% answered that they rarely contacted it. Therefore, most of the students 
in the study were familiar with the metaverse, and only a few had little contact with the metaverse. The school to 
which the students participated in the study belonged had a classroom equipped with three 360° cameras, eight 
laptops, and twenty-four tablet PCs as digital devices for use during class, and students had access to the wireless 
Internet. Also, all students carried smartphones and were available whenever necessary. 


Table 1 
The Ratio of Participant Characteristics of Metaverse 


Experience period Experience type Preference Contact frequency 
Answer % Answer % Answer % Answer % 
Elementary school 43.90 Game 68.29 Very good 43.90 Very often 19.51 
Middle school 7.32 SNS 00 Good 29.27 Often 36.59 
High school 48.78 Learning 26.83 Moderate 21.95 Moderate 31.71 
Etc. 4.88 No so good 4.88 Occasional 9.76 
Dislike .00 Rarely 2.44 


Note. The ratio was calculated using the response results of 41 students finally used for data analysis and is a percentage of the 
ratio of the number of respondents to all respondents. 


Construction of the Learning Program 


Related prior studies were analyzed to develop project activities to explore plants living in school gardens 
using digital twin technology, and learning steps, strategies, and contents were presented. Plant objects placed 
in virtual spaces were searched and identified in advance for the list of plant species found in the actual school 
garden so that students could use them. The species and generic names of plants were presented based on the 
plant list of the National Institute of Biological Resources, and some species that were immature or could not be 
identified due to damage to plants were excluded. 

In order to formation of the learning program in which students directly implement virtual spaces and ex- 
perience inquiry activities, the learning process was decided through consideration of outdoor inquiry programs 
and creative problem-solving processes. In addition, teaching and learning strategies and learning contents were 
presented using the characteristics of the metaverse platform to give students the degree of freedom to lead activi- 
ties and interactions. To construct a plant inquiry learning program using digital twin technology, various research 
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related to scientific problem-solving processes and social interaction were analyzed, and teaching and learning 
strategies were derived by considering outdoor experience activities and research related to the educational use 
of the metaverse. 

The scientific inquiry resulted from modeling scientists’ problem-solving processes from the learning perspec- 
tive (NRC, 2000). Also, the scientific problem-solving process can be divided into the discovery context to generate 
explanatory knowledge and the verification context to confirm the suitability of the generated knowledge (Klaur 
et al., 2002). In the cognitive constructivist perspective, problem-solving processes can be divided into problem 
definition and information discovery, searching for solutions to reach goals, generating and evaluating solutions, 
explaining problem-solving status, and value judgment through communication (Wang & Chiew, 2010). From this 
perspective, NSES suggests that the scientific inquiry process in science classes consists of question definitions from 
current knowledge, brief explanations or hypotheses, simple investigation planning and composition, evidence- 
based explanations, consideration of other explanations, delivery of explanations, and verification of explanations 
(NRC, 2000). Therefore, scientific inquiry activities can be divided into the discovery of scientific problems, tentative 
explanations, and verification of tentative explanations of discovered problems. The discovery process includes 
exploration and generation of information or inquiry problems provided to learners; the verification process includes 
evaluation and explanation of solutions, and recognition of value judgment and communication on inquiry results. 

The plant’s inquiry learning process outdoors is generally divided into before outdoor activity, outdoor activi- 
ties, and after outdoor activity (Braund & Reiss, 2004). On the other hand, inquiry activities in virtual space present 
detailed learning steps such as scientific observation and classification, learning biology, pre-survey, in-virtual 
space exploration, and post-virtual activity organization (Rowe & Humphries, 2004; Subramaniam et al., 2018). In 
addition, it is necessary to construct a relationship between the virtual space and the outdoor activity of plants 
for effective learning (Potkonjak et al., 2016). According to these discussions, plant inquiry using the digital twin 
can consist of discovering a problem or object on scientific inquiry, exploring digital twin technology for scientific 
inquiry, generating the digital twin about outdoor space, and applying scientific inquiry activity. 

The discovery step corresponds to the preparation process for outdoor exploration and digital twin data col- 
lection of plants living in the school garden. In particular, this step is the process of discovering problems or objects 
on scientific inquiry, and students directly establish plans for outdoor inquiry and learn basic knowledge for plant 
observation and identification indoors. Regarding preparation for outdoor plant inquiry, it includes information 
on search paths, habitable plants, and investigation methods in the school garden. Also, regarding digital twin 
data acquisition, available camera tools, filming methods, and filming locations are set in advance. Students are 
allocated to team members who individually contribute to each task for convergent problem-solving that shares 
the same goal in collaborative activities. 

The exploration step corresponds to plant exploration and data acquisition of digital twin outdoors accord- 
ing to the plan established in the previous step. The students discover and identify plants in the school garden 
following the plan, including exploration routes and methods, and record the plants’ characteristics using mobile 
devices. At that time, three-dimensional spatial photography for digital twin data acquisition is performed, and 
image correction is practiced if real-time correction is required about image data. If an error in direction or posi- 
tion occurs after acquiring a three-dimensional image, the immediate correction is more effective for digital twin 
generation than post-correction. Team activities like finding and identifying plants and taking and correcting 
three-dimensional spatial images consist of a divergent problem-solving process for focusing on individually as- 
signed tasks for outdoor inquiry. 

The generation step is exploring digital twin technology students will use to solve scientific problems and create 
the digital twin themselves. According to cognitive constructivist perspectives, cognitively structured experiences 
correspond to essential learning factors (Matthews, 2000). In addition, learning activities using the virtual world 
can improve learning motivation and induce immersion in activities when learners are free to directly adjust and 
control objects (Dede et al., 1999; Psotka, 1995). From this point of view, the main activity of the student is operat- 
ing the software to correct and create virtual space. Artificial intelligence in the platform automatically creates a 
3D digital twin of a school garden when the 3D spatial image data taken in the previous step is uploaded to the 
cloud through the app or web page. 

Anyone can easily create a 3D space, but detecting intentional and unintentional biases contained in spatial 
data and correcting and supplementing it must be performed to prevent distorted results in processing data using 
artificial intelligence (Akter et al., 2021; Byrne, 2021). Therefore, the broad scanning of the created virtual space 
using a mini-map and sky view function to correct image data is performed for discrimination, whether distortion 
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or errors in space. Also, the structure of the space, the description of the characteristics of the plant, excluding the 
shape and location of the plant, and additional photos, videos, web links, Etc., should be added directly by the 
student using the tag function. In addition, virtual tour functions can be used in digital twins, so students can plan 
and organize virtual tour orders and content to share results with others. Therefore, this step composes divergent 
problem-solving activities with individual goals to organize data, such as creating interactive content, explanations, 
and observation resulting in the created 3D digital twin space. 

The application step is a communication process that announces plants living in the school garden using 
digital twins and shares virtual tours for plant introduction creatively organized by student groups. The advantage 
of education using the virtual world is that it shows multiple accessibility and invisibleness (Carvajal et al., 2020), 
and it is necessary to provide a sense of reality by providing interchange opportunities that deal with the virtual 
and real world at the same time for practical educational effects (Potkonjak et al., 2016). Therefore, students must 
present and share the results by group using the results of plant exploration conducted in the exploration stage 
and the data attached to the digital twin space in the creation stage. 

During this process, students create URL links that access digital twins, provide virtual tour functions, and 
share them with students so that other students can experience and compare individual virtual and real worlds. 
In addition, from the perspective of developing students’ creative thinking and transferring learning, investigating 
plant images establishes and presents by utilizing the advantages of digital twins for school gardens. For example, 
the measurement function provided by the digital twin platform can be used to measure the height of trees, the 
expansion width of tree branches, the distribution area of certain plant species, and the speed of expansion of 
plant species habitats over time in cm. 

Unlike the experimental group, the control group, which does not use Digital Twin, uses the results to pro- 
duce information on plants and exploration methods living in the school garden using digital collaboration tools 
such as cloud document tools and presentation production tools and present and communicate by the team. 
Therefore, both groups are active in the same area and plant species, but the experimental group introduces the 
results of the inquiry through the implementation of the virtual world, and the control group presents the results 
as a presentation (Table 2). 


Table 2 
Learning Content and Method between Experimental Group and Contrary Group 


Learning step 


Experimental group 


Control group 


1 2321 


Discovery Basic knowledge learning for plant observation and identification Basic knowledge learning for plant observation 
Planning of plant exploration outdoor activities and identification 
Setting observation paths and methods for plants in the school garden Planning of outdoor plant exploration activities 
Camera preparation for spatial image acquisition Set searching paths and observation methods for 
Setting the method and the point of 3D image acquisition habitable plants 
Exploration Exploration of plants according to the inquiry plan Exploration of plants according to the inquiry plan 
Observation and recording of the plant features with a mobile device Observation and recording of the plant features 
- The identification of found plants (using assistant Al and mini-illustration with a mobile device 
book) - The identification of found plants (using assistant 
- The Acquisition of the 3D image and calibration by mobile device Al and mini-illustration book) 
Generation Operation of the digital twin platform and authoring app Determination of how to present the results of 
Uploading of 3D images to the cloud and Generation of the digital twin plant exploration 
as the virtual space Generation of the presentation contents (descrip- 
Checking and correcting distortion or error in space image tions, photos, and clips of the plant) 
Adding tags in virtual space (plant feature description, photos, images, Exhibition of boards, infographics, Etc, about 
web links, Etc.) inquiry results. 
Organization and presentation of the virtual tour of each group 
Application Sharing of the digital twin URL links Introduction of presentation materials produced by 


Introduction of the virtual tour created by each team 

Presentation of plants living in the school garden using digital twin 
Presentation of ideas of plant inquiry when applying the digital twin 
Self-assessment and peer review among teams 


https://doi.org/10.33225/jbse/23.22.781 


teams 

Presentation of plants living in the school garden 
Presentation of ideas for another plant inquiry 
Self-assessment and peer review among teams 
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Application of the Learning Program 


New innovative technologies can spread quickly through networks, but the personal gap in utilizing technol- 
ogy is so large that simplification and guidance in the methodological aspects of utilizing technology are needed 
(Reimers, 2022). Using advanced technologies in science classes should focus on instrumental use for the efficiency 
of interactive learning rather than on us as a purpose (Chen et al., 2020; Guan et al., 2020), and providing learners 
with an experience of taking the lead can have a positive impact on the defining area of learning (Dede et al., 2017). 

It is necessary to check how it affects students by confirming the degree of change in affective areas such 
as learning motivation, task obsession, and learning attitude of students who have experienced classes in teach- 
ing and learning programs developed according to this perspective. To this end, six classes per two hours were 
organized to be provided to student groups participating in plant inquiry learning programs using digital twins 
and classes to be provided to student groups participating in general plant inquiry classes. Two science educa- 
tion experts and four biology teachers were asked to review the validity of the class guidance plan, including the 
learning process and strategy of the class program, and the content validity index (CVI) was 94%, confirming that 
it was an appropriate composition. 

A total of 12 hours of classes were conducted in the experimental and control groups, and 3 hours were con- 
ducted for each stage according to the class stage leading to discovery, exploration, generation, and application. 
The artificial intelligence app, which will be used to explore plants in outdoor activities, is installed on all students’ 
smartphones before class to be used immediately as soon as the activity begins. In the case of the experimental 
group, to save time in creating a digital twin, all devices were prepared in advance to use the school’s tablet pc and 
360° camera. All students freely used the tablet PC provided by the school and used a laptop for team activities. 


Generation and Operation of Digital Twin 


The generation of the digital twin was performed using Matterport App (Matterport Inc.) to construct a virtual 
school garden space for students. Among various metaverse platforms that use digital twin technology, it is easy 
for users to access and operate for virtual reality generation, and it can be implemented just on mobile phones 
(Ramakrishnan et al., 2021; Sulaiman et al., 2020). Above all, digital twin technology through artificial intelligence 
can accurately express the size and location of objects or spaces present, even if it is highly realistic because it can 
be measured in cm. In addition, teachers or students can add content in the virtual world by tagging interactive 
content such as text, images, and videos, and various functional software like machine learning artificial intelligence 
can be called and used through a link to a web page. 

VR platforms such as Google Street can create virtual spaces, but an additional program must use to replicate 
real-world objects in the cyber world through digital twins. On the other hand, it is optimized for quickly creating 
digital twins for buildings in virtual space. Outdoor environments can replicate if buildings, walls, columns, and 
trees can be used as a reference point for accurately recognizing surrounding objects. In addition, when the user 
uploads a 3D scan image of a target space aimed to replicate to the web cloud, artificial intelligence automatically 
configures an optimized replication space. Above all, the fact that 3D scanning taken with mobile phones and 3D 
cameras can quickly be composed in three dimensions increases educational utilization. In addition, users can 
attach text, images, videos, and web links in the replicated virtual space using the VR interface, and collaboration 
activities are possible through various functions. 

The user needs to create an account for a free digital twin space, and he could use multiple spaces and user 
accounts by subscription. Even if it is possible to post the 3D space to other people by providing a link to a space, 
a paid subscription could be required to edit the many spaces simultaneously. It would help if students prepared 
a camera or mobile phone to shoot target space to compose a digital twin space. However, a 360° camera can use 
an existing camera to capture, and it is also possible through the camera of a mobile phone can install the app. If 
the user uploads the 2D Capture images of the target space to the web cloud, then automatically constructed into 
3D digital twins through Al. Therefore, students need not study and train in the knowledge of VR construction. The 
artificial neural network deep learning model for the digital twin can be used without separate payment; when the 
number of captured images increases, the time required to check the results can be longer. When using a mobile 
phone, the position to focus on is displayed when capturing space in place so that students can hold it with their 
hand and shoot, but a tripod is required to use a 360° camera (Figure 1). 
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Figure 1 
Example of Students’ Virtual Activity Using Digital Twin App 


a © Sea 


geal 


Note. 1) The teacher and students go outdoors to observe, investigate, and identify the plants in the school garden. 2) After the 
cloud upload 360° spatial data through a mobile device, students can automatically create a digital twin for the school garden. 
3) Students can edit the space through an app on a mobile device to operate multi-function; start location, labeling, guide tour, 
tagging, memo, privacy blur, multi-view, and trimming. 4) Students can measure the length, width, and area of tall trees or 
plant communities that are difficult to measure and freely use the tagging function to post plant characteristics and ecological 
features found during the investigation. 


Measurement of Effect on Affective Domain 


Experience in a new form of learning program using digital technology has a positive effect on the level of 
motivation related to student participation (Akour et al., 2022; Stokel-Walker, 2022), and experience activities 
about the nature environment can affect attitudes and learning biology (Ayotte-Beaudet et al., 2017; Wells et al., 
2015). Factors that should be considered for measuring students’ affective domain changes are the level of motiva- 
tion, the intention to participate in learning, the attitude to participate in learning, and the task commitment to 
continue learning (Foronda et al., 2017; Makhija et al., 2018). Attitudes to participate in learning to reflect positive 
or negative perceptions of learning that students are currently performing, and task commitment can provide 
information on the tendency to immerse and continue learning activities they are participating in (Farmer et al., 
2007; Kasperiuniene et al., 2016). 

In this context, the learning motivation test (Keller, 2009) was used to measure the changes in the affective 
domain in learning biology, and researchers checked sub-factors changes; attention, relevance, confidence, and 
satisfaction. The questions for each sub-factor include some reverse questions, including twelve attention-focused 
questions, nine relevance questions, nine confidence questions, and six satisfaction questions. The combined 
result of the question response corresponds to the level of motivation for learning activities. Cronbach’a was very 
high at .91, as a result of calculating by putting it into 20 independent students to confirm the internal reliability 
of the test paper translated into Korean. 21-question learning attitude test (Byeon, 2022) was used to confirm the 
change in students’ attitudes toward biology. The test uses positive and negative words as extremes and contains 
reverse questions to prevent random answers. In addition, a task-commitment test (Byeon, 2022) consisting of 25 
questions was used to confirm the tendency to continue learning biology activities. 
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Analysis of Effect on Affective Domain 


Students responded online before the first class and immediately after the end of the last class to each survey. 
After response data collection, the response results were statistically analyzed to confirm the effect of applying the 
learning program before and after the classes between the experimental group and the control group. Learning 
disposition was presented before the learning motivation, learning attitude, and task commitment test for learning 
biology, and the results of the pre-post response of each test paper calculated the average value and performed 
variance analysis. In addition, a chi-square test was conducted on metaverse and learning biology-related tenden- 
cies to determine whether there are differences in student factors that can affect the group effect. 

It is necessary to confirm the main effects of the experience of indoor and outdoor parallel biology class 
programs conducted by both groups on students’ affective domain to analyze the effect of classes conducted on 
student groups in this study. Therefore, a linear regression analysis was conducted by setting the group of students 
participating in the class activities as dummy variables for the main effect analysis and setting the pre-test results as 
independent variables and the post-results as dependent variables. Afterward, the group effect was confirmed by 
conducting a one-way distributed analysis of learning motivation, learning attitude, and task obsession using the 
group asa factor to confirm the effect of the class program applied independently to each group on the affective area. 


Research Results 
Learning Disposition of the Metaverse and Biology 
It is necessary to analyze students’ learning tendencies of biology and the metaverse before confirming the 
effect of the application of the class program on students. Therefore, independent samples t-tests were conducted 
using student responses to the interest, the awareness of the importance, and the preference of related occupation 


of the metaverse and biology learning (Table 3). 


Table 3 
Learning Disposition Difference Between Two Groups 


EG CG 
Learning disposition ne: t p 
M SD M SD 
Interest in the metaverse learning 4.35 15 4.00 95 -1.309 920 
Awareness of the importance of the metaverse learning 4.10 85 3.71 96 -1.361 = .735 
Preference for the metaverse-related occupation 3.40 1.05 2.57 1.21 -2.343 341 
Interest in the biology learning 4.05 945 3.90 1.04 -466 = 312 
Awareness of the importance of the biology learning 3.90 1.02 3.86 1.01 -.135 901 
Preference for the biology-related occupation 3.00 1.23 2.95 1.28 -126 179 


Note. EG = experimental group, CG = control group. 


There was no significant difference in the analysis results of students’ responses on the learning propensity 
and interest in the metaverse and biology learning between the experimental and control groups. Therefore, it 
was necessary to identify individual factors that may affect participation in the learning program developed in 
this study. According to the result of the chi-square test conducted to check the effect of student factor, it was 
following for the presence or absence of experience in the metaverse learning x2 = 1.003, df= 1, p =.317, and asa 
result of the mainly used metaverse type ¥? = 3.864, df= 4, p = .425. In addition, the chi-square test results for the 
acquisition method of the metaverse information were y2 = 6.113, df= 5, p = .295, and the results for the contact 
frequency of the metaverse information were y? = 2.621, df = 4, p = .623. The acquisition method of the biology 
information was 2 = 3.578, df = 3, p = .311, and the analysis results for the contact frequency of biology informa- 
tion were y? = .739, df = 4, p = .946. Therefore, there was no difference in the group regarding the metaverse and 
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biology experiences and information between the experimental and control group students. Therefore, it can be 
regarded as a homogeneous group with little difference between the experimental and control groups in the 
metaverse and biology learning context. 


Main Effect of the Application of the Learning Program 


It is necessary first to check the main effect of whether the indoor and outdoor parallel class program applied 
to the experimental group and the control group causes a change in the affective domain of learning biology to 
analyze the group effect of digital twin application. The main effect analysis was conducted through linear regres- 
sion analysis of 41 students in the two groups’ biological learning motivation, learning attitude, and task-obsession 
test results before and after class. As a result of conducting a linear regression analysis using the group as adummy 
variable and setting the pre-test result as an independent variable and the post-test result as a dependent variable, 
the variance analysis results for the generated model were statistically significant (Table 4). 


Table 4 
The Regression Model on the Student's Affective Domain 


95% Cl 
Model Estimate SS p 
LL UL 
Learning motivation @ 864 .108 645 1.083 <.001 
Learning attitude ° 627 107 409 844 <.001 
Task commitment ° 971 123 12) 1.220 <.001 


Note. Total N = 41. Cl = confidence interval; LL = lower limit; UL = upper limit. 
* Dependent variable = post motivation, R? = .678.° Dependent variable = post attitude, R? = .560. ‘ Dependent variable = post 
task commitment, R? = .672. 


According to the affective area’s pre and post-test results according to the application of classes by group, 
learning motivation increased from pre-result (M = 122.83, SD = 17.46) to post-result (M = 143.05, SD = 20.53). In 
addition, the learning attitude increased from pre result (M = 87.22, SD = 12.02) to post result (M = 97.93, SD = 
11.82), and the task commitment increased from pre result (M = 88.98, SD = 10.25) to post result (VM = 100.71, SD 
= 13.47). Through the main effect analysis results, the increase in the level of the affective domain after class was 
statistically significant compared to before the class program was applied in this study. Therefore, it can be inter- 
preted that the experience of the indoor and outdoor combined biology inquiry class positively affects students’ 
motivation, attitude, and task commitment to learning biology. 

The regression model coefficients derived from the analysis showed significant pre-test to post-test, so it can 
be interpreted that both groups have main effects from the application of indoor and outdoor parallel classes. In 
addition, since the group variable was found to be statistically significant, it was confirmed that there was a differ- 
ence in effects according to the group (Table 5). 


Table 5 
Coefficient of Model about The Change of Student's Affective Domain 


Measurement Model B SE B t p 
Learning motivation Group -14.002 3.738 -.345 3.745 .001 
Learning motivation 864 108 135 7.972 <.001 

Learning attitude Group 6.969 2.553 -.298 -2.730 01 
Learning attitude 627 107 637 5.831 <.001 
Task commitment Group -7.146 2.498 -.268 -2.861 007 
Task commitment 971 123 738 7.870 <.001 
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Group Effect by the Application of Learning Program 


This study developed a learning program using digital twin technology so teachers and students can eas- 
ily explore outdoor plants and have various interactive experiences. In addition, students were divided into an 
experimental group and a control group to confirm the impact of the developed learning program on students 
were applied. The learning program using a digital twin was applied to the experimental group, and a program 
without a Digital twin was applied to the control group to confirm the group effect. The average for each group 
was calculated by conducting the test of students’ affective domain according to the experience of the teaching 
and learning program of the experimental group and the control group (Table 6). 


Table 6 
Mean of Pre-test and Post-test for Affective Domain between Students Groups 


Experimental group Control group 
Measurement Pre-test Post-test Pre-test Post-test 
M SD M SD M SD M SD 


Motivation of learning biology 123.50 16.58 150.80 17.83 122.19 1864 135.67 20.58 


Attention 42.80 5.29 51.85 6.53 42.57 6.28 47.05 6.96 

Relevance 32.00 4.41 38.30 5.12 30.76 4.93 34.62 5.56 
Sub-factor 

Confidence 26.85 5.62 33.75 5.21 27.57 5.33 30.38 5.78 

Satisfaction 21.85 2.89 26.90 3.01 21.29 3.91 23.62 3.67 
Attitude of learning biology 89.30 13.96 10280 10.47 85.24 9.77 93.29 11.37 
Task commitment of learning biology 90.40 9.78 105.75 11.04 87.62 10.74 95.90 14.06 


The mean of the experimental group's post-test results increased more than the pre-test results on the moti- 
vation for learning biology at 27.30, while the mean of the comparison group’s result increased to 13.48. For both 
groups, the mean of the post-test results was higher than the pre-test results, confirming that the instructional 
program applied to the students could positively affect the affective domain. Therefore, the change in each sub- 
factor of the motivation was calculated in detail to confirm the cause of the difference in the level of motivation 
for learning biology between the two groups. The experimental group's mean change of attention factor was 
higher at 9.05 than the control group's change of 4.48, and in the relevance factor, the experimental group's mean 
change was higher at 6.30 than the control group's 3.86. In addition, the confidence factor change was higher in 
the experimental group, 6.90, than in the control group, 2.81. The change of satisfaction factor in the experimental 
group at 5.05 was higher than control group at 2.33. In the overall factor of motivation, it indicated a relatively high 
increase in the experimental group's mean. 

In addition to learning motivation, the mean change in the experimental and control groups for attitude and 
task commitment of learning biology was calculated. The mean of the attitude of learning biology in the experimen- 
tal group increased to 13.50, higher than the control group's change of 8.05. In addition, the experimental group's 
mean change of task commitment of learning biology was higher at 15.35 than the control group’s change of 8.29. 

In both the experimental and control groups, the post-test results increased compared to the pre-test results. 
These changes can be interpreted as an increase in students’ positive perception and tendency to continue learn- 
ing about biology, including learning motivation, due to the experience of direct inquiry activities in the natural 
environment and learner-led activities. On the other hand, the fact that the change in the experimental group 
was more significant than that of the control group can be interpreted as positively affecting the sub-elements of 
learning motivation by creating, manipulating, and controlling the virtual world directly through digital twins. In 
addition, positive perception and continuity of learning biology were relatively increased due to students’ partici- 
pation and immersion increased using a new digital technology called Digital Twin. 

In both groups, the post-test results about the motivation of learning biology were higher than the pre-results, 
and the average for each detailed area also increased significantly afterward. In other words, it means that both 
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the experimental and control groups can positively change their motivation for learning biology through the ex- 
perience of biology teaching and learning programs for plant exploration. In addition, the experimental and the 
control groups showed higher post-test results than the pre-test results on the attitude and task commitment of 
learning biology, so the teaching and learning programs experienced by students can positively affect learning 
biology attitudes and task commitment. Also, since the change in the experimental group was more significant than 
the control group, the experience of classes using digital twins could be positive for students’ learning motivation, 
perception of biology, and intention to continue learning. Therefore, One-way ANOVA was conducted to confirm 
whether these effects were statistically significant (Table 7). 


Table 7 
One-way ANOVA for the Change of Students’ Motivation for Learning Biology 


Measurement SS df Ms F p 
Motivation of learning biology Between group 10744.590 3 3581.530 10.468 <.001 
Within group 26688.105 78 342.155 
Attention Between group 1159.045 3 386.348 9.722 <.001 
Within group 3099.845 78 39.742 
Relevance Between group 677.050 3 225.683 8.922 <.001 
Within group 1972.962 78 25.294 
Sub-factor 
Confidence Between group 594.885 3 198.295 6.575 .001 
Within group 2352.395 78 30.159 
Satisfaction Between group 387.924 3 129.308 11.138 <.001 
Within group 905.588 78 11.610 
Attitude of learning biology Between group 3446.566 3 1148.855 8.721 <.001 
Within group 10275.495 78 131.737 
Task commitment of learning biology Between group 3893.627 3 1297.876 9.742 <.001 
Within group 10391.312 78 133.222 


Note. SS = Sum of square, MS = Mean of square. 


According to the result of ANOVA, motivation of learning biology and sub-factors showed that the post-test 
results in both groups increased statistically significantly compared to the pre-test, and there were also differences 
between groups. In other words, it means that the change in the pre-and post-test results of the experimental 
group increased at a significant level compared to the control group. Accordingly, the effect size within the group 
was Calculated using the pre and post-test results to confirm the effectiveness of the class program applied in this 
study on learning motivation. 

The effect size on the experimental group was Cohen's d = 1.238, and Cohen's d = .936 in the control group, 
and both groups showed an effect size of a high level of learning motivation. Therefore, the biology class program 
applied to both groups positively affected the motivation of learning biology within groups. Consequently, experi- 
encing outdoor life exploration classes that cross indoor and outdoor can improve students’ attention, relevance, 
confidence, and satisfaction with biological learning and positively affect changes in learning motivation. 

The statistical analysis of variance in the attitudes and the task commitment for learning biology showed 
that the post-test results in both groups increased significantly compared to the pre-test results, and differences 
between groups existed. These results mean a statistically significant difference in the attitude toward learning 
biology of the experimental group and the change in task commitment for learning biology compared to the 
change in the control group. Accordingly, the effect size within the group was calculated to confirm the effect of 
the class program applied in this study on the attitude and task commitment to learning biology. 

As a result of calculating the effect size using the pre and post-test of the two groups, the effect size about 
the attitude of learning biology within the experimental group was Cohen's d= 1.094, and the comparison group 
was Cohen's d = .759, showing a very high-level effect. In addition, Cohen’s d was calculated using the results of 
the task commitment test for learning biology, the experimental group was Cohen's d = 1.472, and the control 
group was Cohen's d= .662, indicating that the teaching and learning program at each group effectively improved 
task commitment. Consequently, the biology class program applied to both groups positively affects students’ 
affective domain. 
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Since it has been confirmed that the effect size of the class applied to the group is significant, it is necessary 
to statistically check whether there is a difference in the degree to which the class applied to each group provides 
a positive effect on the students’ affective domain. Therefore, to confirm the effect size of the class program ap- 
plied independently to the two groups, the partial eta square value was calculated using the pre-test result as a 
covariate and the post-test result as a dependent variable. The partial eta square value of the learning motivation 
for learning biology was n? = .270, and the group effect size was very high. Therefore, analyzing the differences 
between groups and the effect size confirmed that the experimental group could induce a much more positive 
effect than the control group. Also, the partial eta square value about the attitude of learning biology was n* =.164, 
and the task commitment of learning biology was n? = .177. Therefore, it confirmed that the experimental group 
could induce a much more positive effect on the attitude and task commitment than the control group according 
to the analysis results of the differences between groups and the effect size. 

In conclusion, affective areas such as learning motivation, attitude, and task obsession for learning biology 
changed significantly in the control group that applied learners’ self-directedness and indoor and outdoor life 
exploration classes. Digital twin classes were more positive in the experimental group than in the control group. 


Discussion 


Exploring various organisms outdoors is essential for learning biology, offering advantages that indoor activities 
cannot provide. However, outdoor activities for teachers come with limitations in the real world. As a result, virtual 
reality technology, which transports exploration sites to virtual spaces, has been proposed as a way to learn about 
organisms. With the advancement of digital technology, the potential for the educational use of the metaverse is 
growing. However, successful learning becomes possible only when effective teaching and learning strategies are 
employed to properly utilize these technologies for scientific education (OECD, 2022). From this perspective, this 
study presents the following considerations for using virtual reality in learning. 

First, the virtual world should be constructed so that students can feel more realistic. Traditional virtual real- 
ity learning activities require professional skills because they rely on computer graphics implemented through 
rendering, so some users feel less realistic (Cobb, 2009; Garrison, 2016). In addition, from the student's point of 
view, virtual world control is complex and requires passive use, reducing the opportunity for interaction, which 
can negatively affect the affective domain (Fauville et al., 2021; Nadler, 2020). On the other hand, the more similar 
the virtual environment is to reality, the more realistic students perceive the virtual space, and the more effective 
learning experience can be induced (Kwon, 2020). Therefore, to use immersive virtual space for learning, it needs to 
provide the same level of space and experience as possible. Unlike the rendering-based space generation method 
using computer graphics programs, this study maximized the sense of reality by replicating real space and objects 
using a digital twin platform that constructs space with artificial intelligence through 3D photography. It was shown 
that there was a very high level of effect in the affective domain change of the experimental group participating 
in this study, and the effect on the level of learning motivation and task commitment was greater than that of the 
control group. These results can be attributed to the fact that students could measure and interact with the actual 
size of objects and spaces in the virtual space in the experimental group that provided the exact size and shape 
of 3D space, objects, and locations reflecting depth and distance. 

Second, it is necessary to provide freedom for learners to organize the virtual world and interact with various 
interactions easily. The level of motivation for learning participation may vary depending on the degree of freedom 
given in learning activities and the initiative of activities, and motivation for learning participation is the driving 
force for continuing learning (Karatas et al., 2017). In particular, in science learning activities, providing learners 
with self-directedness and freedom can positively affect learning motivation and learning (Dede et al., 2017). In 
addition, learning participation attitudes reflect students’ positive or negative perceptions of the learning they are 
currently performing, and task commitment can provide information on their tendency to immerse themselves 
in and continue their learning activities (Farmer et al., 2007; Kasperiuniene et al., 2016). Unlike activities that only 
utilize passively provided virtual reality, this study provided the initiative for experimental group students to 
directly create digital twins in the virtual world and use them by adding various interaction objects. In addition, 
students can immediately check and compare with the real one in the school garden during the action in a virtual 
reality space, and it can arise to form a positive perception of living things. The experimental group’s results that 
created and utilized the digital twin themselves showed that providing students with a degree of freedom through 
a change in attitude toward learning biology and task commitment was necessary. 
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Summarizing the above discussions, using the virtual world as an alternative to outdoor biology inquiry 
activities can lead to learners’ motivation to participate, a positive perception, and a tendency to continue learn- 
ing biology when learners can construct and interact freely. To this end, various digital twin platforms should be 
provided so anyone can easily construct and operate a realistic virtual space. It is very encouraging that the effect 
on the affective domain of the experimental group that experienced digital twin-based exploratory learning ac- 
tivities in this study is more favorable than the control group, which means that indoor and outdoor exploratory 
activities linked to the virtual world can be effective in changing students’ perception of actual learning biology. 


Conclusions and Implications 


This study used digital twins to create a realistic virtual world, apply it to outdoor inquiry classes, and investi- 
gate the effect on students’ affective areas. Researchers could present several conclusions and implications based 
on various research findings and discussions. 

First, an outdoor inquiry class for learning biology using the virtual world with a high sense of reality using 
digital twins was able to organize. Ironically, the biggest problem in learning using the virtual world is its unrealism. 
In other words, learners who are users must passively consume the virtual space that developers provide. Even im- 
ages perceived in the real world, different levels of graphic space, and limitations that cannot be compared to the 
real world reduce students’ sense of presence. However, in this study, the digital twin platform allowed students to 
create virtual replication spaces in school gardens with simple digital devices and compare them to the real world. 
Therefore, this study confirmed the possibility of implementing hybrid learning that can be converged between 
the real world and the virtual world, which is aimed at the educational environment of the future digital society. 

Second, providing students with a sense of realism and initiative in activities can positively affect learning 
biology in the real world through virtual world activities. Another problem in learning using virtual reality is pas- 
sivity and isolation. Virtual spaces created by experts cannot be expanded or modified by users, and new content 
can be added only by developers. However, the digital twin platform allows anyone to modify or update virtual 
spaces by sharing and adding content. In addition, team activities can resolve individual isolation caused by learn- 
ing using virtual reality, and interactions between virtual and real worlds increase opportunities for interaction 
between humans. Increasing learners’ initiative and interaction can positively affect the motivation, perception, 
and learning continuity of learning biology about the real world through virtual world activities. 

In this study, an outdoor inquiry program for learning biology using the digital twin focuses on allowing 
students to freely replicate, organize, and manipulate the real world with simple digital device manipulation. 
Therefore, we can propose the following implications. 

First, anyone can easily create and utilize a virtual world to expand to a variety of learning biology fields. The 
learning program using the digital twin platform in this study focuses on learners creating dynamic learning con- 
tent that is more than just 3D space and continuously interacting with the real world. Therefore, it can be used to 
implement new ways and ideas in learning biology that broadens the boundaries of indoor and outdoor learning. 
For example, digital twin data acquired continuously at a particular point can monitor changes in plant populations 
over time. In addition, visualized 3D and multimedia information about ecosystems in different regions can be 
continuously updated and used for long-distance project learning. Also, various uses such as simulating dangerous 
experiments, individualized classes for students with different experiments, and saving time through repetition of 
the same experiment will be possible. 

Second, anyone anywhere can use the virtual world just using a mobile device, helping bridge the digital 
divide. The digital divide in learning using virtual reality is occurring at national, regional, school, and home levels, 
and without professional equipment and expertise, students could not even create a virtual world. These problems 
cause differences in opportunities for teachers and students to build and experience a virtual world, and they act as 
problems that widen the digital divide. However, the virtual space replication activities and educational utilization 
proposed in this study can be used at the student level without professional knowledge, only on mobile devices 
with internet access. Therefore, if anyone can choose the target space they want to explore, quickly replicate it in 
the virtual world, and use it for learning, they can close the digital gap. 
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PERCEPTIONS OF PRIMARY 
PRE-SERVICE TEACHERS IN 
THE UTILIZATION OF PLANT 
IDENTIFICATION APPS AS 
EDUCATIONAL TOOLS 


Peter Paul Canuto 


Introduction 


Teaching botany among students has always been challenging. How- 
ever, rapid technological advancement and the integration of computer 
science research paved the way for the development of mobile learning that 
supports and promotes plant profiling and raising plant awareness. One such 
advancement to support learning about plants is automated plant identifi- 
cation apps that can be readily downloaded and installed on smartphones. 
Utilizing plant identification apps combined with the curation of digital plant 
images and image recognition technology is already becoming functionally 
significant (Boho et al., 2020; Jones, 2020) as it easily supports our ability to 
identify plants (Waldchen et al., 2018). 

Sadly, there has been a drop in the public’s knowledge and interest in 
studying plants, which may result from urbanization and the industrial and 
technological revolution (Burke et al., 2022). There is also an observed lack 
and underrated interest in plants among students despite their ecological 
and economic relevance (Weigelt et al., 2022). Some studies indicated that 
most students preferred to study animal species compared to plants (Balas 
& Momsen, 2014; Wandersee & Schussler, 1999). Students also find plant 
identification more complex and tedious than animal identification (Wang, 
2017). These factors may relate to the concept of “plant awareness disparity 
(PAD)”, proposed by Parsley (2020). This concept expounds the individual's 
tendency to ignore plants in their surroundings, where they visualize plants 
as a scenic aggregation of green mass secondary to animals (Parsley, 2020; 
Parsley et al., 2022). 

PAD is a modified term of “plant blindness” by Wandersee and Schussler 
(1999) since the former term is often perceived as associated with visual 
disability and discriminatory towards persons with disabilities as a whole 
(McDonough MacKenzie et al., 2019; Parsley, 2020). Students experience 
PAD depending on their attention or regard for plants, attitude or feeling of 
learning about plants, knowledge or recognition of the relevance of plants, 
and relative interest between plants and animals (Parsley, 2020; Parsley et 
al., 2022). This occurrence may be due to one’s perception and how the 
brain functions (Strgar, 2007). Students perceive that they have more similar 
characteristics to animals, such as movement and physical resemblance, than 
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Abstract. Plant identification apps make 
learning about plants more convenient. 
This study explored the participants’ 
perceptions of using three plant 
identification apps, PlantNet, PictureThis, 
and LeafSnap, as potential educational 
tools. Problems experienced, differences 

in perceptions, and the participants’ most 
preferred apps were also determined. 
Through purposive sampling, the study 
engaged 162 primary pre-service teachers 
in the Cordillera Administrative Region 
(CAR), Philippines. Data were collected 
through a developed questionnaire and 
analysed quantitatively. The questionnaire 
was reliable with an identified single 
component for perception. Participants first 
explored and used the apps to identify local 
plants, thereafter, responding through an 
online questionnaire. Results showed that 
participants strongly perceived the apps as 
engaging, helpful in plant identification, 
easy to browse, providing details, effective 
as emerging tools, and significant for 
scientific literacy, except for consistency of 
results. There were significant differences, 
but with small effect sizes, indicating 
negligible differences in the perceptions of 
male and female participants regarding 
the apps’ consistency of results and 
effectiveness. Weak internet connection 
was the primary issue affecting the apps’ 
utilization. The pre-service teachers 
preferred LeafSnap over PictureThis and 
PlantNet. Conclusively, this study affirmed 
the potential of the apps for students 
learning about plants, further supporting 
their feasibility as emerging educational 
tools. 

Keywords: educational tools, plant 
identification apps, PlantNet, PictureThis, 
LeafSnap, primary pre-service teachers 
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plants (Hoekstra, 2000). Students strongly (Prokop & Fanéovicova, 2023) and automatically prefer animals first to 
plants since they belong to the animal kingdom (Flannery, 2002). 

Plants are vital players in biogeochemical processes, such as carbon and nitrogen cycles, contributing to our 
approaches related to global climate and climate change. With the importance of plants, schools and teachers 
should keep their instruction manageable because they are significant to our understanding of ecosystems (Pany 
et al., 2019) as it underpins food webs (Hill, 2022). Because of these factors, teachers must engage and exert effort 
to raise interest in botany and promote student plant awareness and appreciation (Pany et al., 2019; Strgar, 2007). 
One way to achieve these goals is through plant identification applications (apps) that enable the students to gain 
knowledge of plant species (Zhu et al., 2017). 

There has been a significant increase in the use of plant identification apps over the years (Baker, 2023). With 
the correct plant identification app, individuals can contribute to citizen science initiatives, engage with nature, 
and determine which plant species are invasive or dangerous (Airhart, 2023). Nonetheless, these apps must be 
meticulously utilized to effectively impact science literacy and citizen science. It is therefore recommended to use 
plant identification apps that have reputable sources, such as government or university-affiliated sites (Hill, 2022). 
It is noteworthy to use apps thoroughly tested and evaluated for their accuracy in identifying plants (Partel et al., 
2021). In the academe, teachers must use these apps based on reputable sources and expert systems to ensure 
their contribution to students’ plant knowledge (Wang, 2017). 


Literature Review 
Botany in the Primary Science Curriculum and Local Plants 


The present Kindergarten to Grade 12 (K to12) Science Curriculum of the Philippines aims to develop scientifi- 
cally, technologically, and environmentally literate students. These literacies enable them to become productive in 
the community, solve complex problems, be innovative, make well-informed decisions, communicate effectively, 
exhibit scientific attitudes and values, and become responsible custodians of nature. This curriculum employs a 
spiralling progression of scientific concepts in Chemistry, Biology, Physics, and Earth and Space Science, leading 
to a deepened understanding of the students as they progress to higher levels. In particular, formal teaching of 
Science as a discipline starts at the Grade 3 level. From Kindergarten to Grade 2, science learning focuses on ex- 
ploratory activities and arousing students’ curiosity about the natural world to foster basic scientific knowledge 
and skills development. These conditions invoke students’ readiness for formal Science learning (Department of 
Education, 2016). 

The Philippine K to 12 Science Curriculum also underscores localized learning of plants found in the com- 
munity. It reiterates that students need to learn about their local plants to understand food and nutrition, bio- 
diversity, sustainability, medicinal and traditional uses, and strengthen plants’ cultural awareness and heritage 
(Department of Education, 2016). The Cordilleran region is an agricultural community situated in mountain ranges. 
Due to its location, temperatures, humidity, and weather patterns differ across the region, making it ideal for 
various vegetable and fruit crops, including ornamental cut flowers. Some examples of locally grown vegetable 
and fruit crops include but are not limited to mongo or munggo (mung bean), camote (sweet potato), broccoli, 
cauliflower, okra (lady’s fingers), kangkong (water spinach or swamp cabbage), lettuce, pechay (napa cabbage or 
snow cabbage), ampalaya (bitter gourd) fruit, tomato, string beans, squash, ginger, bell pepper, carrots, gabi (taro 
or taro root), radish, potato, camote (sweet potato), and cassava (Philippine Statistics Authority, 2022). Likewise, 
ornamental cut flowers include gladiola (sword lily), asters, gerbera daisies, coloured and white calla lilies, Shasta 
daisies, chrysanthemum, agapanthus (Lily of the Nile or African lily), anthuriums, statice (sea lavender), baby's 
breath, alstroemeria (Peruvian lily), milflores (hydrangea), lisianthus, carnations, and snapdragons (Cabreza, 2023) 
to name a few. 

The foundations of learning about plants are laid at the primary level. Teachers formally introduce the stu- 
dents to the natural world, associating it with their basic scientific knowledge and skills. In this stage, students are 
exposed to the incredible diversity of plant life on Earth's ecosystems. They will learn to identify different types 
of plants, their parts, growth patterns, adaptations, modes of reproduction, and habitats. The plants’ functions in 
the environment, their benefits, and proper care are also emphasized. This leads the students to develop recogni- 
tion and appreciation for the vast array of colours, shapes, and sizes of plants (Department of Education, 2016). 

Consequently, having innovative and creative teaching aids and supplementary resources is a must among 
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teachers. These resources help the teachers instil a sense of wonder, curiosity, respect for the natural world, 
scientific literacy, environmental awareness, plant appreciation, and an understanding of the interconnections 
between plants, humans, and the environment. For these reasons, primary school teachers need a spectrum of 
educational materials that can be used inside and around the community. Using plant samples, slide presentations, 
and video clips is practical in teaching botany. Nonetheless, using interactive, digital, and mobile learning will 
contribute further to students’ learning about plants, allowing them to explore more plants in their environment. 
Hence, the reason for exploring pre-service teachers’ perceptions of the potential of plant identification apps as 
educational tools. As pre-service teachers, they are gaining and developing knowledge and skills for facilitating 
learning by using a broad range of teaching methodologies, innovative and creative resources, and utilizing 
educational technologies that they could apply in their future teaching profession (Commission on Higher Educa- 
tion, 2017). Allowing them to explore the plant identification apps and understand their perception may engage 
the pre-service teachers to effectively employ these apps during their practicum or future teaching endeavours. 


Plant Identification Apps 


Automated plant identification apps are mobile apps that use photo recognition algorithms and software 
programmed to identify plants (Pimm et al., 2015; Sachar & Kumar, 2021). Typically, the user takes a photograph 
of the whole or any of the parts of the unknown plant and then uploads it to the app's system. Once the photo- 
graphs are uploaded, the app compares them to its database, analyses them, and then provides output or lists 
of closely possible results (Sachar & Kumar, 2021; Waldchen & Mader, 2018). For apps that provide lists of closely 
possible results, the first plant presented is considered the highest possible identification of the plant. In contrast, 
the others are suggestions allowing the user to select the most accurate result (Schmidt et al., 2022). 

The app’s databases may come from various sources, such as crowdsourced observations, remotely sensed 
images, or specimen collections from museums or herbariums worldwide (Pimm et al., 2015; Sachar & Kumar, 
2021). Conveniently, the leaf is the most photographed plant part due to its presence throughout the year (Sa- 
char & Kumar, 2021), regardless of the seasons. Characteristics used for comparisons and analysis between the 
uploaded photograph and the app’s databases include the leaf’s colour, shape, texture, and venation (Sachar & 
Kumar, 2021; Waldchen & Mader, 2018). Other plant parts commonly used by these apps for identification include 
stems, bark, flower, or fruits (Jones, 2020; Rzanny et al., 2019; Schmidt et al., 2022). 

The selection of plant identification apps, PlantNet, PictureThis, and LeafSnap, involved in this study was 
primarily based on Hill’s (2022) long-term evaluation. From 2018 to 2021, Hill (2022) and their students tested 14 
apps concerning the apps’ accuracy in identifying plants. In 2021, they tested eight apps up to 90 to 140 times 
to identify plants at the W. J. Beal Botanical Garden, Michigan State University. They have photographed plants 
of flowering broadleaf ornamental species and several kinds of flowering and vegetative weeds categorized by 
broadleaf weeds, grass or grass-like weeds, and a seedling winter annual weed. Their study also included the 
botanical garden’s usual agricultural and turf weeds. In the end, as of the fall of 2021, Hill (2022) ranked Pictur- 
eThis as the top-performing app based on the app’s comparative percentage of correctly identifying the plants. 
The PlantNet app ranked second, while LeafSnap ranked fourth. PlantStory, the third in rank, was not chosen for 
this study since it was newly evaluated by Hill (2022) in 2021, unlike the other three apps. PlantStory is also best 
described as a marketplace for plants, focusing on commercial rather than educational purposes. 


PlantNet 


PlantNet or Pl@ntNet is an excellent software for identifying plants quickly. It also provides consistent and 
accurate identifications of plants compared to others. The app functions through the support of the Agropolis 
Foundation. It was developed as a project through sponsorship of four French research institutes consisting of 
the National Institute of Agricultural Research (INRA), the French National Research Institute for Sustainable De- 
velopment (IRD), the French Institute for Research in Computer Science and Automation (INRIA), and the French 
Agricultural Research Centre for International Development (CIRAD), including the Tela Botanica Network (Baker, 
2023; Waldchen et al., 2018). 

PlantNet provides guides on how to use the app’s features (Baker, 2023). Using filters, it can identify plants 
using photos of leaves, flowers, fruits, or bark. It also uses the Global Positioning System (GPS) and Map option 
to accurately identify plants (Baker, 2023; Waldchen et al., 2018). It relies on quality image recognition and user 
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involvement (Baker, 2023). It can share photo documentation for research purposes but does not share data with 
third parties for advertisement (Airhart, 2023). PlantNet also supports multiple languages (Baker, 2023) and does 
not contain pop-ups and ads requiring users to pay for other features (Airhart, 2023). However, it does not give 
detailed information, which may be lacking for others (Airhart, 2023). 

Based on the evaluations of Hill (2022), PlantNet emerged as the second top-performing app. It achieved 55% 
accuracy in identifying seedling winter annuals and almost the same performance as PictureThis in identifying 
vegetative grass-like weeds. Hart et al. (2023) determined the apps’ accuracy at 86.6%. On the other hand, based 
on the works of Hill (2022), CNN journalist Baker (2023) and her colleagues conducted a related evaluation and 
concluded that PlantNet is the most accurate plant identification app among the apps they have tested. Thus, 
they highly recommend PlantNet as the best app for most people. Another evaluation of the app by The New York 
Times Wirecutter journalist Airhart (2023) and her colleagues found that PlantNet is simple, quick, and easy to use 
compared to other apps, ultimately picking the app as the best among the apps they have also tested. Alternately, 
Otter et al. (2020) found that the app had 47% accuracy in identifying toxic plants, making it second in rank. 


PictureThis 


PictureThis is a plant identification app developed by Glority Global Group Limited, an artificial intelligence 
(Al) resource based in China. It uses Al deep learning technology to search its database and identify plants (Parkins, 
2019). Their official website shows that the app can identify more than 10,000 plant species. PictureThis provides 
topics related to the identified plant, such as full description, toxicity, informative videos, and natural history in 
considerably more detail (Airhart, 2023). Aside from these, it has a “plantpedia’, a digital plant encyclopedia con- 
taining helpful information, such as ideal habitat, soil requirements, planting season, and pest control. The app 
allows users to share pictures and answer plant-related questions through its worldwide social community and 
forum sections. It also has a “flower map” feature that allows users to tag plants in their locality (Parkins, 2019). 

Hill (2022) concluded that PictureThis is the top-performing app. Hill (2022) and her students found that 
PictureThis ranked the top-performing plant identification app with 67% accuracy among the 130 plant samples. 
It maintained its number-one rank four years in a row. Further, it was the favourite of Hill's (2022) students due to 
its high accuracy and ease of usage. Though it maintained its top rank, it presents a low accuracy of 47% when 
identifying vegetative grass or grass-like weeds compared to other categories. Accordingly, a closer inspection 
of the grass structures, such as floret and ligule, is necessary and is mostly not seen in the whole plant picture. 
Similarly, Schmidt et al. (2022) found that PictureThis resulted in a high correct identification of 81.36% genus 
and 67.84% of species of combined leaf and bark structures among 55 tree species. With these high accuracy 
results, they will likely recommend the app for their students’ use. Another study by Otter et al. (2020) concluded 
that the app was the best among the apps they have used, with a 59% accuracy rate in identifying toxic plant 
species. On the contrary, Airhart (2023) and her colleagues found that PictureThis is less user-friendly than the 
other tested apps. It is also too easy to unintentionally subscribe to paid extra features and advertisements pop- 
ping up from the app. 


LeafSnap 


LeafSnap was developed by researchers from the Smithsonian Institution, the University of Maryland, and 
Columbia University (Waldchen et al., 2018). It compares the plant’s picture to its database of over 9000 images 
(Kress et al., 2018). The app retrieves photos on its database similar to the uploaded one. However, the user has 
the last say in whatever species they think best fits the findings (Waldchen et al., 2018). LeafSnap has a broader, 
more accurate advanced identification function that can identify the plant using photos of the whole plant, 
leaves, or flowers. It presents caring guides and reminders for plants which can be helpful for the users. Users can 
also create plant photo collections and data sharing (Baker, 2023). Moreover, it provides background information 
about the species and its geographic location (Kress et al., 2018). 

Compared with other apps, LeafSnap was identified as the fourth-best app by Hill (2022). It achieved an ac- 
curacy of more than 40%. It was able to have better performance in identifying flowering ornamentals. Waldchen 
et al. (2018) indicated that LeafSnap achieved 96.8% among 184 tree species, making it to the top five and an 
overall recognition rate of 73% accuracy. Hart et al. (2023) measured the apps’ accuracy at 86.9%. Alternatively, 
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Kress et al. (2018) observed a 10% to 100% accuracy variance among the tree species samples. Nonetheless, they 
have noted that LeafSnap is continuously undergoing improvement using deep learning. Moreover, Airhart (2023) 
observed that using the app presented challenges since operating could be more intuitive. It also presents pop- 
ups and advertisements that may cause accidental subscription of the user. 


Research Gap to be Filled 


Even with the growing number and use of plant identification apps, only a few studies (Wang, 2017; Yujuan 
et al., 2021) were conducted on the feasibility and capability of mobile plant identification apps as educational 
tools. Most studies conducted on these apps focused primarily on their features, accuracy, user-friendliness, and 
downsides (Airhart, 2023; Baker, 2023; Hart et al., 2023; Hill, 2022; Kress et al., 2018; Otter et al., 2020; Partel et 
al., 2021; Schmidt et al., 2022; Waldchen et al., 2018). There is still a lack of recognition of these apps’ potential 
for science teaching. This prompted the researcher to explore the prospect of using plant identification apps for 
learning about plants which are deemed beneficial to raise plant awareness and recognition and impede PAD 
among the students. Instead of accuracy, this study centred on examining and understanding the perspective 
of future primary school teachers about their views and experiences, thereby acknowledging the capabilities of 
the apps to be used in science education. 


Research Aim and Research Questions 


This study aimed to explore the potential of three plant identification apps, PlantNet, PictureThis, and 
LeafSnap, as emerging and interactive educational tools for students learning about plants. Mainly, it aimed to 
determine the perceptions of primary pre-service teachers in using the apps as a probable teaching aid in upper 
primary grade levels and their significance for supporting the development of scientific literacy. It also aimed 
to determine the problems experienced by the participants in using the apps. The differences between the par- 
ticipants’ perceptions according to gender and their most preferred app were also determined. Moreover, the 
following research questions guided the conduct of this study: 

1. What are the primary pre-service teachers’ perceptions regarding using plant identification apps as 
educational tools? 
2. Are there significant differences among the participants’ perceptions according to gender? 
What are the problems experienced by the participants while utilizing the apps? 
4. Which app is the most preferred one by the participants? 


oo 


Research Methodology 
Research Design 


The researcher employed a quantitative approach to gather and analyse the perceptions in using the 
three plant identification apps for learning about plants and the challenges experienced by the participants. 
A descriptive survey was used to understand the characteristics of the participants (Anastas, 2000), leading to 
the exploration of the participants’ insights and struggles concerning the apps. Meanwhile, inferential statistics 
was used to compare the differences (Kuhar, 2010) among the participants’ perceptions of utilizing the apps for 
teaching about local plants. This exploratory study focused on primary pre-service teachers’ perceptive views and 
ordeals regarding their future use of plant identification apps. The pre-service teachers were enrolled in Bachelor 
of Elementary Education (BEEd) from four state and private universities in the Cordillera Administrative Region 
(CAR), Philippines. The BEEd leads explicitly to a teaching profession at the primary level. The apps utilized by 
the participants involved three plant identification apps that were selected based on the long-term study of Hill 
(2022). The participants used the apps to identify locally grown vegetable and fruit crops and ornamental plants 
around their community. The participants’ overall perceptions and differences according to gender, problems they 
encountered, and most preferred apps were determined using an online survey questionnaire provided through 
email. The study was conducted throughout the two semesters of the academic year 2021 - 2022. 
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Participants 


The study involved 162 BEEd pre-service teachers. They comprised second to fourth-year undergraduate stu- 
dents, involving 46 (28%) males and 116 (72%) females. The number of participants was based on those pre-service 
teachers who volunteered, consented to this study, fully utilized the plant identification apps, and thoroughly 
answered the online questionnaire. Preliminary to data gathering, request letters were sent to a total popula- 
tion of 176 primary pre-service teachers enrolled in four universities. However, only 162 out of 176, or 92% of the 
participants, fully agreed, had used the apps and entirely responded to the questions. The remaining 8% of the 
participants either withdrew their participation or submitted incomplete responses. A 92% of positive responses 
resulted most likely due to the follow-up reminders (Fincham, 2008; Menon & Muraleedharan, 2020) undertaken 
by the researcher, attaining enough data to represent the population. 

Furthermore, the participants were determined through purposive sampling, enabling their selection based 
on the structure of this study (Creswell et al., 2011). The curriculum design, objectives, and career opportunities of 
the degree, which calls for the development of qualified teachers with specializations in content and pedagogy 
for primary education (Commission on Higher Education, 2017), guided the selection of the BEEd pre-service 
teachers. The participants’ gender is the only variable considered in the study. The year level of the participants 
was not included since some were classified as irregular students. In the context of the country’s universities, an 
irregular student is a student who is enrolled with less than the prescribed number of course subjects or units in a 
given semester required by the curriculum of the degree program. It may also pertain to students attending classes 
across different year levels of the same program to comply with the required number of units for the degree. For 
the irregular students involved as participants, it is difficult to determine their specific year level; hence, the year 
level is excluded. 


Instrument 


The researcher primarily developed the questionnaire with four sections. The first section of the questionnaire 
gathered the profiles, indicating gender, year level, and classification as either regular or irregular students, from 
the primary pre-service teachers. The second section comprised seven question items, as presented in Tables 4 
and 5, items 1 - 7, regarding the participants’ perceptions of the apps. These question items will be rated through 
a 5-point Likert scale measured with 1 = Strongly Disagree (SD), 2 = Disagree (D), 3 = Neutral (N), 4 = Agree (A), and 
5 = Strongly Agree (SA). The third section consisted of a checklist with five question items enumerated in Table 6, 
items 1 -— 5, regarding the problems they encountered while using the apps. This section allowed the participants 
to select the given options more than once. A selection of the three apps presented in Table 7 is the last section of 
the questionnaire prompting a singular response from the participants to choose the best app to use in teaching 
significant lessons related to local plants. The question items were integrated into an online questionnaire created 
using Google Forms. 


Reliability and Factor Analysis 


Content validation of the questionnaire was first requested from three faculty members with expertise in 
Biology teaching. Some of the statements were then paraphrased based on validations. Thereupon, pilot testing 
was conducted involving 30 primary pre-service teachers from a state and a private university in the same region, 
excluding these universities during the primary data gathering. As a result, the questionnaire was found reli- 
able, with a determined coefficient of reliability of .952 using Cronbach's alpha. Subsequently, exploratory factor 
analysis was done on the seven question items regarding perception, ruling out questions concerning problems 
with multiple responses and the most preferred app. The correlation matrix in Table 1 shows that the Pearson cor- 
relation coefficient of each question item ranges from .409 to 1. Most items showed a high correlation with each 
other, considering a correlation coefficient of .60 and above, indicating that the items belonged to the same factor. 
Those items with a correlation coefficient of below .60 showed moderate correlation, such as in the case of Item 
3 correlated with Items 4, 5, 6, and 7. There were no negative and weak correlations identified among the items. 
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Table 1 


Correlation Matrix of Question Items Regarding Perceptions 


Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 
Item 1 1.000 
Item 2 631 1.000 
Item 3 608 544 1.000 
Correlation Item 4 510 653 581 1.000 
Item 5 681 709 409 454 1.000 
Item 6 576 769 445 466 727 1.000 
Item 7 538 115 487 634 703 806 


Kaiser-Mayer-Olkin (KMO) and Bartlett's test of sphericity was also done. The KMO was determined as .859 
(greater than .70), indicating enough items to produce a factor. Likewise, Bartlett's test is significant (y(21) = 133.65, 
p=.001) at 5%, showing that the variables were correlated highly enough to provide a reasonable basis for factor 
analysis. Based on the total variance explained shown in Table 2, only the first component has an eigenvalue greater 
than 1. This result stipulated that there is only one possible factor. The cumulative percent showed that only one 
factor accounts for 66.54% of the variance. 


Table 2 
Total Variance Explained 


Initial Eigenvalues Extraction Sums of Squared Loadings 
Component 
Total % of Variance Cumulative % Total % of Variance Cumulative % 
1 4.658 66.539 66.539 4.658 66.539 66.539 
2 806 11.521 78.060 
3 582 8.315 86.375 
4 370 5.292 91.667 
5 228 3.258 94.924 
6 207 2.964 97.889 
7 148 2.111 100.000 


Similarly, the component matrix shows that only one factor was extracted, as indicated in Table 3. Item 2 
has the highest correlation value among all the items, indicating its strong association with the only component. 
Meanwhile, Item 3 has the lowest correlation value, though still indicating a strong association with the component. 
The remaining items also disclosed strong associations based on their correlation values. Since there is only one 
component, all items were correlated with only one factor. Considering all the question items, the commonality 
among them is that they all relate to using and evaluating the mobile apps used for plant identification, referenc- 
ing the user’s perceptions. 
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Table 3 


Component Matrix of the Items 


Component 
Items 
1 
‘em 2: | found the mobile apps helpful for identifying plants 899 
‘em 7: The mobile apps are significant for supporting scientific literacy 876 
‘em 6: | found the mobile apps as an effective emerging tool for learning about plants 852 
em 5: The mobile apps provide significant details about the plants 830 
‘em 1: | found the mobile apps engaging for plant exploration 192 
‘em 4: There is consistency of results among the three mobile apps 145 
‘em 3: | found it easy to browse within the mobile apps’ interface 697 


Procedures 


First, a letter of permission indicating the study’s objectives and the participants’ contribution was sent to the target 
primary pre-service teachers through emails. Then, consent forms asking for the participant's involvement in the study 
were also sent to those who positively gave their permission. After the participants’ approval, they were instructed to 
download and install the three plant identification apps, PlantNet, PictureThis, and LeafSnap, from either Play Store 
for Android systems or App Store for iOS systems, depending on the model and compatibility of their smartphones. 
Afterwards, the participants were asked to use the plant identification apps to identify local vegetable and fruit crops 
and ornamental plants they liked. The plants to be identified may or may not be familiar to them. They may take photos 
of the whole plant, its flowers, leaves, of barks. The participants were allowed to use the apps to identify unlimited local 
plants. This condition allowed the participants to be immersed in using the apps and familiarize themselves with their 
features. However, they were only allowed to finally choose five plants for the purpose of this study. They also had to 
take note of the scientific name of the identified plant by the apps for comparison of result consistency. For monitor- 
ing and record purposes, the participants were also instructed to take a screenshot of the apps’ results and send it 
through the researcher's email. After using the apps, a link to the Google Form containing the online questionnaire 
was forwarded to the participants for them to answer. Results were then tabulated using spreadsheets. The conduct 
of this study upholds the practice of scientific inquiry and the ethical and professional standards in doing research. 
The contact information, especially the participants’ email addresses, was gathered through their professors’ referrals 
and was strictly used for communication purposes relevant to this study. Consent forms were asked from the partici- 
pants’ emphasizing their voluntary participation. The online questionnaire was sent and monitored personally by the 
researcher. The submitted responses and their tabulated data were compiled and kept confidential. 


Data Analysis 


A statistical limit of weighted mean was used to analyse the perception of the utilization of the plant iden- 
tification apps for learning about plants among the participants. The statistical ranges consist of 1.00 - 1.80 = 
Strongly Disagree (SD), 1.81 - 2.60 = Disagree (D), 2.61 - 3.40 = Neutral (N), 3.41 - 4.20 = Agree (A), and 4.21 -5.00= 
Strongly Agree (SA). Frequency count, percentage, and ranking were used to determine the problems experienced 
by the participants in using the apps and to determine their most preferred teaching tool among the three apps. 
Moreover, a t-test was used to test if there were significant differences in the level of perceptions of the use of the 
apps between the male and female primary pre-service teachers. 


Research Results 
Table 4 shows that the primary pre-service teachers have strong perceptions towards the three apps after 


using them, except for one item. They strongly agree that the apps were effective emerging tools for learning 
about plants, followed by helpful apps for identifying plants. The participants expressed the same level of agree- 
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ment regarding the apps’ features in providing significant plant details and engaging for plant exploration. There 
is also a strong agreement relating to the apps’ ease of browsing and significance for supporting scientific literacy. 
Notably, they have a lower level of agreement on the apps’ consistency of results. 


Table 4 
Perceptions of the Primary Pre-service Teachers in Using the Plant Identification Apps as Probable Educational Tools 


Perceptions M SD Level 
1. | found the mobile apps engaging for plant exploration 4.43 0.802 Strongly Agree 
2. | found the mobile apps helpful for identifying plants 4.46 0.723 Strongly Agree 
3. | found it easy to browse within the mobile apps’ interface 4.35 0.784 Strongly Agree 
4. There is consistency of results among the three mobile apps 3.91 0.866 Agree 
5. The mobile apps provide significant details about the plants 4.43 0.738 Strongly Agree 
6. ee apps as an effective emerging tool for learning 4.49 0.766 Strongly Agree 
7. The mobile apps are significant for supporting scientific literacy 4.31 0.774 Strongly Agree 


There are variances in the participants’ perceptions when grouped according to their gender, as Table 5 
indicates. Though there is an overall strong agreement concerning the three plant identification apps, there are 
peculiar differences between the participant’s perceptions at the .05 critical level. There are no observed significant 
differences in most perceptions between male and female pre-service teachers. On the other hand, results indicate 
significant differences between the male and female participants’ perceptions towards the apps’ consistency of plant 
identification results (p = .020) and effectiveness as emerging tools for plant learning (p = .043). However, further 
analysis (not shown in the table) conveyed that the effect size, determined by Cohen's, was d = 0.49, indicating 
a small effect size between the participants’ perceptions of the apps’ consistency of plant identification results. 
Similarly, the measured Cohen's was d = 0.47, indicating a small effect size regarding the effectiveness of the apps 
as emerging tools for plant learning. Thus, the effect sizes between the male and female participants concerning 
the two perceptions are negligible, even if they are statistically significant. 


Table 5 
Differences in the Participant's Perceptions in Using the Plant Identification Apps as Probable Educational Tools According to 
Gender 


Males Females 
Perceptions ee of t p 
M SD M SD 

1. | found the mobile apps engaging for plant ex- 4.46 0.520 4.41 0.697 95 .325ns 146 
ploration 

2. | found the mobile apps helpful for identifying plants 4.35 0.587 4.50 0.496 7 -1.166ns 247 

3. | found it easy to browse within the mobile apps’ 4.37 0.594 4.34 0.628 85 .183ns 855 
interface 

4. There is consistency of results among the three 3.65 0.810 4.02 0.695 7 -2.377* .020 
mobile apps 

5. The mobile apps provide significant details about 4.30 0.572 447 0.530 80 -1.302ns 197 
the plants 

6. | found the mobile apps as an effective emerging 4.28 0.741 4.58 0.507 71 -2.062* 043 
tool for learning about plants 

7. The mobile apps are significant for supporting 4.17 0.680 4.36 0.563 76 -1.342ns .183 


scientific literacy 
Note: * = significant (p < .05); ns = not significant (p > .05) 
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Aside from perceptions, Table 6 presents the problems encountered by the participants in using the three 
apps. Overall, the main problem encountered by the primary pre-service teachers relates to a weak internet con- 
nection that affected the apps’ functions. This is followed by the consistency of results, the need for a subscription, 
the compatibility of the apps with their smartphones, and the complexity of the apps’ interface. Further analysis 
showed that three problems attained the same ranking from both genders except the remaining two. Male and 
female pre-service teachers experienced the same problems concerning poor internet connectivity, the consistency 
of plant identification results of the three apps, and the need to fully subscribe to use the apps’ features. Males, 
however, experience more complications in using than the compatibility of the apps. Females struggled more with 
the compatibility with their smartphones than the difficulty of the apps’ user interface. 


Table 6 
Problems Experienced by the Participants in Using the Plant Identification Apps 


Male Female Overall 


Problems experienced 
f % Rank f % Rank f % Rank 
1. The plant identification results are not 13 19 2 34 21 2 47 21 2 
consistent among the three mobile apps 
2. The mobile apps are not compatible with 9 13 5 15 9 4 24 11 4 
the smartphone’s specifications 
3. The mobile apps’ interfaces are complex 10 15 4 12 8 5 22 10 5 
to utilize 
4. Subscription is needed among the mobile 12 18 3 26 16 3 38 16 3 
apps 
5. The mobile app functions are affected by 24 35 1 73 46 1 97 42 1 


a weak internet connection 


Regarding preferences shown in Table 7, LeafSnap emerged as the primary pre-service teachers’ most pre- 
ferred plant identification app. The other apps, PictureThis and PlantNet, ranked second and third, respectively. 
Correspondingly, the top preferred app between males and females varies. Males highly preferred PictureThis 
compared to LeafSnap by the females. Males secondarily chose LeafSnap, while females selected PictureThis. Both 
genders identified PlantNet as their least preferred among the three apps. 


Table 7 
The Most Preferred Plant Identification App Among the Participants 


Male Female Overall 


Plant identification apps 
f % Rank f % Rank f % Rank 
PlantNet 10 22 3 26 22 3 36 22 3 
PictureThis 22 48 1 37 32 2 59 37 2 
LeafSnap 14 30 2 53 46 1 67 41 1 


Discussion 


The overall perception regarding the plant identification apps of the primary pre-service teachers showed 
strong agreement, indicating the apps’ prospective use as educational tools. This highlights the emerging and effec- 
tive use of mobile learning through smartphones. The importance of teaching students about plants as fundamental 
to Biology (Pany et al., 2019; Strgar, 2007) may further increase due to using these apps. Aside from citizen science 
and scientific literacy, the students’ technological literacy will be advanced. The fact that the participants find the 
apps engaging for exploring and helpful in identifying local plants reinforces the recognition and awareness of 
plants in our surroundings. These apps aid active and interactive learning outside of the classroom. Wang (2017) 
deduced that these apps contribute to students’ outdoor engagement, affecting their learning and behaviour. 
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The participants’ strong agreement on the ease of using and browsing the apps agrees with Airhart’s (2023) 
general observations of some apps. Most apps are user-friendly and contain menus and tabs that are easy to 
navigate. These features cause hassle-free browsing by the users. The essential details the apps provide about the 
identified plants are beneficial. As Airhart (2023) indicated, this feature gives users relevant information about the 
plant. This may enable the users to acquire knowledge and further explore the plant's growth and development, 
structure, habitat, and palatability. The details can then contribute to understanding the plant’s aesthetic and 
biological features, as Wandersee and Schussler (1999) pointed out. 

The strong perception of both primary and pre-service teachers on the potential of the apps to be used as an 
effective emerging tool for learning about plants and their significance in supporting scientific literacy affirms the 
usefulness of plant identification apps in education. This study supports and further contributes to the works of 
Wang (2017), implying the feasibility of plant identification apps in science instruction. It also supports Yujuan et 
al. (2021), who recommend using plant identification apps to train autonomous learning of students about plants. 
Wang (2017) concluded that these apps significantly improved the students’ science learning and attitudes and 
caused better learning satisfaction than traditional teaching methods. With these apps, students may be able to 
identify plants easily (Waldchen et al., 2018), nurture plant knowledge (Zhu et al., 2017), and to further recognize 
and appreciate plant's role in the biosphere (Hill, 2022; Pany et al., 2019). This may then lead to an increase in 
students’ plant awareness and reduce their PAD. However, it must be noted that teachers must choose apps that 
are extensively evaluated (Airhart, 2023; Baker, 2023; Hill, 2022; Kress et al., 2018; Otter et al., 2020; Schmidt et al., 
2022; Waldchen et al., 2018) and uses reputable sources and expert systems (Hill, 2022; Partel et al., 2021) to ensure 
positive students’ learning. 

Regarding the consistency of results of the three apps to which the participants considerably agreed, it con- 
firms the findings and evaluations of identified studies (Airhart, 2023; Hart et al., 2023; Hill, 2022; Kress et al., 2018; 
Otter et al., 2020; Schmidt et al., 2022; Waldchen et al., 2018) that not all three apps provide the same plant names. 
The variances in the results imply the limitations of the apps and their accuracy in identifying plants through im- 
ages. Despite the advancement of automated learning, Al, and deep learning, circumstances affecting the apps’ 
accuracy rate are still unclear (Hart et al., 2023). 

Though there are observed significant differences in the perceptions of the male and female pre-service 
teachers towards the apps’ consistency of results and effectiveness as emerging tools for plant learning, their mea- 
sured effect sizes are small. This implies trivial differences between the male and female perceptions, having low 
effect sizes and negligible differences. In this case, there are no practical variances in all of the pre-service teachers’ 
perceptions according to gender. There are no differences in how the male and female participants perceive the 
apps’ engaging capability for plant exploration, helpfulness in plant identification, ease of browsing, consistency of 
results, providing details, effectiveness as potential educational material, and relevant support for scientific literacy. 

A weak or poor internet connection is the participants’ main problem in using the apps. This condition is 
primarily affected by the country’s performance in providing internet services, the cost of internet services and 
subscriptions (Rosales, 2023; Samaniego, 2023), telecommunications infrastructure, and the geographical environ- 
ment (Salac & Kim, 2016). Due to poor connection, the participants may have difficulties accessing the features 
of the apps, uploading images, and loading results. Since these apps use online databases, a strong internet con- 
nection is advisable. 

Problems relating to the consistency of results among the apps were also experienced. Accordingly, the consis- 
tency and accuracy in the plant identification results may be affected by several factors, including the diversity and 
differences of leaves as the basis for identification (Sachar & Kumar, 2021; Zhang et al., 2020), the bias and validation 
between the tests and the sample plant images which may result to unreliable species labels (Jones, 2020; Rzanny 
et al., 2019), the short span of production of flowers and fruits among plants (Perera & Arudchelvam, 2021), the 
lack of data presentation and depiction of other organs of the plant (Boho et al., 2020), the quality of the images 
uploaded (Hill, 2022), and the rareness of some species (Jones, 2020). This is especially true in the region since most 
data stored in the apps’ databases are collected from Western countries, affecting the identification of local plants. 

Similar to Airhart’s (2023) evaluation, subscribing to the apps and the sudden pop-up ads were encountered 
as well by the participants. In the free versions, some apps provided a minimum number of photos to be uploaded 
within the day and prevented some features from being accessed, limiting the use of apps. Users must purchase 
and subscribe to the apps to fully enjoy their system. The participants’ struggles with the applications’ compatibility 
with their smartphones and the complexity of the apps’ user interfaces are technological and reliant on the apps’ 
software and operating system. Updating the apps, the incorporation of Al (Joly et al., 2016) and deep learning 
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, 


(Kress et al., 2018), developers’ coding techniques and algorithms (Waldchen et al., 2018), scientific institutions 
contributions and validations (Joly et al., 2016), and image processing (Yanikoglu et al., 2014) may be some of the 
factors that affected the said problems. 

Overall, the pre-service teachers prefer LeafSnap the best over the other two apps despite ranking as the 
fourth-best app, as Hill (2022) established. However, Waldchen et al. (2018) still pointed out that the app had high 
accuracy. They may have ranked LeafSnap as the best due to its ability to provide plant care reminders and create 
albums of plant collections, as Baker (2023) presented. The second most preferred app is PictureThis, in contrast 
to Hill's (2022) and Otter’s et al. (2020) inferences and Schmidt's et al. (2022) high recommendation of the app 
for students’ use. Though Airhart (2023) found the app less user-friendly and full of pop-up advertisements, the 
pre-service teachers may have ranked it second due to its features, such as a plant encyclopaedia and local plant 
tagging, as Parkins (2019) described. PlantNet, the participants’ least preferred app, counters the apps’ second-best 
ranks by Hill (2022) and Otter et al. (2020). It does not also contend with Baker's (2023) and Airhart’s (2023) high 
regard for the app. This may be due to the lack of plant information provided by the app, as reported by Airhart 
(2023), affecting the participants’ view of the app. Nevertheless, it is important to note that the participants’ prefer- 
ences do not depend solely on the three apps’ accuracy and consistency of results. Instead, it generally involves 
the participants’ perceptions of the apps’ features and potential as educational tools, including their experiences 
using the three apps. 


Conclusions and Implications 


Utilizing plant identification apps is feasible for classroom learning, as affirmed by the results of this study. 
The pre-service teachers strongly agreed on the apps’ engaging characteristics, usefulness in plant identification, 
ease of browsing, providing information, effectiveness for learning about plants, and relevance in supporting sci- 
entific literacy. The participants showed lower perceptions regarding the apps’ consistency of results. It was also 
determined that there were no significant and practical differences between the male and female participants’ 
perceptions of the apps. This is despite the drawbacks they have experienced while using the apps, mainly affected 
by weak internet connection. Ultimately, the pre-service teachers selected LeafSnap as their preferred app over 
PictureThis and PlantNet. Conclusively, these plant identification apps can provide opportunities to reinforce and 
support instruction. Almost all students nowadays have smartphones and can access the internet. Various plant 
identification apps are free and readily available to identify plants quickly. Integrating these apps into teaching 
can be advantageous from an educational aspect and is very important for learning about plants. The use of these 
apps is recommended and timely, concerning the advancement of technology, its impact on science education, 
and in strengthening plant awareness for the conservation of natural resources. However, the apps should undergo 
a thorough and methodical evaluation to ensure their suitability for students’ learning and congruence with the 
objectives of the lessons. 

The findings of this study contributed to the enrichment of relevant literature. It also offered a glimpse into 
the participants’ perceptions about the potential of plant identification apps for mobile learning. It allowed the 
exploration of these apps as emerging educational tools, therefore, presenting the participants’ confirmation of 
the apps’ possible classroom use for those educators who may be planning to do so. The non-comparison between 
the apps’ accuracy of results is a limiting factor in the study. Listing the results and analysing their precision and 
consistency was not employed. Another factor relates to the number of apps involved in the study. There were 
only three apps explored in comparison to the earlier evaluations. Also, the participants involved were associated 
with primary grades where learning about plants is considerably at the beginning stage. Lastly, the study focused 
on the pre-service teachers’ perceptions and did not consider student involvement. With these, future researchers 
should explore the effectiveness of plant identification apps in student learning other than perception. Adding 
similar and relevant apps is also suggested. Furthermore, an extensive study should be conducted on the apps’ 
practicality as a teaching aid and their impact on science education. 
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CORRELATION BETWEEN 
ENERGY METAPHOR AND 
EMPATHY TENDENCY 


Mustafa Erdemir, 
Sebnem Kandil ingec 


Introduction 


The cognitive learning theory regards learning as a cognitive process 
and focuses on mental operations (encoding, storing, and retrieving informa- 
tion) to make sense of the occurrences in the environment. In parallel with 
the cognitive learning approach, the dominant view of the concept of the 
metaphor (the metaphor is the art of speaking, embellishing speech, and elo- 
quence) has gradually begun to change. In addition to the art of eloquence, 
the view that the metaphor is the product of a mental process and that the 
focus of the metaphor is not a language but a mental mechanism has gained 
prominence. Lakoff (1993) emphasized that the metaphor is not only the art 
of eloquence but also reflective of the way of thinking. 

While authors, poets, and artists were interested in metaphors and 
empathy in classical times, today, different fields of science have shown 
interest in the concepts. Artists reflect the mental models they developed 
based on their empathic perceptions of their works. Artworks are a product 
of the artist’s emotional, cognitive, and empathic state. Authors and artists 
have tried to put themselves in the shoes of what they are trying to depict, 
believing that empathetic understanding is the key to good writing and 
practical art (Smith, 2012). The person looking at a work of art internalizes 
the artist’s thoughts and feelings through empathy. As how internalization is 
made varies by person, a work of art evokes different feelings and thoughts 
in everyone. Metaphors and empathy are a way of reflecting on one’s mental 
models of the world. Metaphors and empathy are similar concepts, and in 
situations of creating the metaphor and empathy, relationships and similarity 
patterns are used (Swan & Riley, 2012). 

Individuals utilize different features and connections when creating 
metaphors, which makes metaphors unique to the individual. Empathy, on 
the other hand, is about how the other person is perceived and internalizing 
his behavior cognitively or emotionally. Empathy is understanding the other 
person's feelings, perspectives, or situations and experiencing the other per- 
son's emotional state (Swan & Riley, 2012). When a person uses the metaphor 
and practices empathy, they acknowledge another person’s mental models, 
knowledge, and experiences through mental processes. The metaphor is 
a way of reflecting perceptions, concepts, and thoughts through mental 
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Abstract. Creating a metaphor for the 
concept of energy can contribute to the 
determination of the mental structuring of 
the concept of energy. Pre-service teachers’ 
metaphors about the concept of energy 
were classified according to traditional and 
scientific fields. The metaphors produced 
by the mathematics and science teacher 
candidates for the concept of energy 
consisted of structural and abstract 
concepts, approximately 60% of them were 
the continuity of life and 40% were science 
categories. While pre-service mathematics 
teachers mainly produce metaphors 

for biology, pre-service science teachers 
produce metaphors mainly for physics. 
Empathic tendency scores and levels of 
pre-service teachers were determined. 
Pre-service teachers have a high level of 
empathic disposition and predominantly 
see the other person's problem and emotion 
as their own. There was no quantitatively 
significant difference between traditional 
metaphor types and empathic tendencies. 
It is understood that the pre-service 
teachers have high empathic tendencies, 
and mostly in the you stage, the concepts 
related to the continuity of life are primarily 
used in developing metaphors related to 
the concept of energy, and they tend to 
produce structural metaphors. Metaphor 
and empathy will contribute to the 
determination of mental models and the 
establishment of a connection between the 
fields of science. 
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processes. Morgan (1998) defined metaphors as a way of thinking and seeing in perceiving the world. Empathy, 
on the other hand, is trying to understand the thoughts and feelings of another person and seeing the world from 
their point of view. Empathy is the ability to make inferences by decoding another person's thoughts, feelings, and 
behaviors (Swan & Riley, 2012). Metaphors and empathy are mental processes at the end of which the external 
projection of internalized situations is presented. The metaphor is a way of reflecting perceptions, concepts, and 
thoughts through mental processes, while empathy is acknowledging the feelings and thoughts of another person. 
In creating metaphors, mental models are developed based on existing information while empathy is the practice of 
perceiving the feelings and thoughts of another person and acknowledging them through unique mental models. 


The Metaphor 


Mental models (scientific models) are defined as a set of meaningful metaphors, and one way of projecting 
mental models is by using metaphors. Lakoff and Johnson (1980) stated that metaphorical concepts are the es- 
sence of scientific thinking and that without them, little can be understood beyond our direct physical experience. 
Mental models help us understand the surrounding phenomena around us by making meaningful connections 
with and extending our prior knowledge. Thinking metaphorically enables anew phenomenon to be understood 
by relating it to previous experiences (mental models). Conceptual metaphors represent the relationship mental 
models and are created by connections such as relationships and analogies. Additionally, metaphors serve as the 
interface between mental models and are an element of the mental mapping mechanism, which shapes thinking 
(Beasley & Waugh, 1996). 

It is difficult to directly observe and measure an abstract concept. The concept of energy is appropriately de- 
fined within the confines of science, but these definitions do not apply to all fields. For example, students think that 
the definition of energy in biology is different from that in physics. Lakoff and Johnson's (1980, 1999) conceptual 
metaphor theory presents an ideal method through which to teach this topic (Amin, 2009; Dreyfus et al., 2014; 
Lancor, 2014; Scherr et al., 2012). Conceptual metaphor theory plays an important role in understanding abstract 
concepts, being a cognitive theory arguing for the largely metaphorical nature of the way we understand the world 
and that most conceptual structures are built based on metaphors that help us make sense of the world through 
the familiar (Lakoff & Johnson, 1980, 1999). A new experience or concept is constructed by establishing relations 
between mental models, and metaphors are used in understanding a concept or field based on other concepts or 
experiences. Metaphorical language is essential for expressing and understanding abstract ideas, and conceptual 
metaphor theory provides a way for researchers to gain insight into how students understand abstract concepts 
such as energy (Lancor, 2015). With the metaphor, a new concept, phenomenon, or object is instinctively associ- 
ated with the important features of the concepts, phenomena, and objects that exist in our minds, making the new 
concepts, phenomena, and objects understandable. This helps to interact with and make sense of the environment. 

When we conceptualize an experience or an idea, we make sense of the experience or idea by picking its most 
important parts and comparing them with what already exists in the mind. This mental process creates metaphors 
by pairing phenomena, objects, and concepts otherwise difficult to understand with other concepts that have 
common characteristics, thus facilitating learning and recall by making them understandable. Metaphorization 
is explaining a concept or phenomenon by drawing an analogy with another concept and phenomenon (Oxford 
et al., 1998). The metaphor enables complex and abstract aspects of reality to be understood in more concrete, 
familiar, and easily imaginable terms (Refaie, 2001; Semino, 2008). Metaphors also contribute to eloquence, are 
effective in persuading and influencing others, and facilitate strengthening communication and building intimacy 
between the speaker and the listener (Semino, 2008). 


Empathy 


The Turkish Language Association (2018) defined empathy as the ability of an individual to put themselves 
in the shoes of another individual in terms of emotion and thought. Empathy is a human trait, a psychological 
mechanism, that enables individuals to establish and maintain social relationships and to take an evaluative stance 
toward the vantage point of aesthetics and epistemology. It is also a behavior that enables individuals to make logi- 
cal inferences from moral and social mistakes by empathizing and reasoning with and drawing lessons from them. 
Earlier, empathy was addressed from the vantage point of aesthetics and epistemology. Aesthetic empathy refers 
to individuals’ spontaneous projection of feelings and thoughts while observing objects, whereas epistemological 
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empathy has been used by artists and poets as a means to achieve their goals (Gulseren, 2001). 

Today, empathy is defined as putting oneself in the other person's shoes, seeing the similarities and differences 
in the other person's ideas, and respecting and tolerating differences. Respecting others’ opinions is important for 
establishing healthy communication and relationships. Denis (1999) stated that those who are good at building 
positive relationships generally empathize with people more easily. Empathy is taking the place of the other person 
and possessing and internalizing the same emotions and ideas as them. Baron-Cohen (2003) defined empathy as 
the urge to identify the thoughts and emotions of another person and respond to them with an appropriate emo- 
tion. Reacting by understanding the other person facilitates healthy social relations and achieving common goals. 

The literature distinguishes between two types of empathy, namely cognitive and emotional (Brems, 1989; 
Hoffman, 1977; Rogers, 1980; Shantz, 1975; Strayer, 1987), and the type of empathy practiced changes by a situa- 
tion (Gladstein, 1983). Cognitive empathy is the ability to accurately perceive or imagine what another person is 
thinking and feeling by ascribing mental states to them. This mental process is sometimes referred to as the theory 
of mind or perspective (Davis, 1983; Hogan, 1969). Emotional empathy is the emotional response of individuals 
who put themselves in another's place. It refers to sharing emotional responses, for example, feeling distressed in 
response to the other person’s pain or feeling happy in response to their joy. Hoffman (2008) defined emotional 
empathy as the ability to identify with the emotions of others and to understand what the other individual is feeling. 
Emotional and cognitive empathy are equally important in communicating with others, perceiving what they are 
thinking and feeling, relating to them emotionally, and allowing them to share their thoughts. There is a consensus 
in the literature that these two elements are inseparable (Feshbach, 1975; Greenberg et al., 1993; Greenson, 1960; 
Katz, 1963; Schafer, 1959; Strayer, 1987). 


Metaphors, Empathy and Teaching Energy 


In everyday life, mental models are unwittingly used to understand the environment and describe phenom- 
ena. Metaphors and empathy are abstract concepts involving mental modeling. Mental models (scientific models) 
include analogies or metaphors (Aubusson et al., 2006) and are often constructed through analogical reasoning 
(Collins & Gentner, 1987). Often mental models are verbalized through metaphoric and empathetic responses. 
These responses serve to activate, elaborate, or modify mental models as in understanding something (Hestenes, 
2006). Therefore, understanding the language used to communicate the mental model provides insight into this 
model (Lancor, 2015). In teaching, with the help of metaphors and empathy, students’ mental models of the sub- 
jects can be identified, in light of which the course content can be restructured and processes suitable for these 
mental models can be developed. 

Metaphors constantly change due to the physical and cultural interaction with the environment, which 
enables the development of mental models and paves the way to establishing new connections with different 
fields. Metaphors can be used as learning outcomes that aim to identify students’ mental models and thoughts on 
the subject. Since metaphors and empathy are projections of mental models, they can be used to restructure the 
curriculum and teaching methods used and determine learning outcomes. In addition, examples of empathetic 
utterances can be used in the in-class interactions between students and teachers. 

Students can use metaphors as a tool to describe and understand their environment and practice empathy 
to build healthy communication and relationships. While creating a metaphor, it is possible to expand an existing 
model or create a new one by establishing relationships, similarities, and commonalities between mental models. 
They stated that there is a relationship between mental models and metaphors (Duit, 1991; Hestenes, 2006), and 
empathy is the basis for how we understand someone else’s mind and how we reflect our own mental states or 
create a mental model (Fonagy et al, 1991). Empathy is basically metaphorical (Swan & Riley, 2012), empathy 
and metaphor are formed as a result of models using mental information, emotions, thoughts and experiences. 
Metaphors function as a pathway to empathy (Smith, 2012). Familiar terms are used when generating metaphors 
and practicing empathy. Individuals likely empathize with victims similar to themselves (family members), which 
is called “familiarity bias” (Hoffman, 2000). 

Empathy and metaphors can be utilized in the interactions between students and teachers in the classroom 
environment. The teacher can empathize with what the students are thinking and feeling with the help of emotional 
and cognitive expressions such as worry, feedback, understanding, and reaction. This use of empathy makes it easier 
for the teacher to meet the students’ needs. Additionally, teaching-learning models can be adjusted in light of the 
information about the mental processes and structures students utilize to create metaphors and practice empathy. 
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To eliminate the complexity of the concept of energy and establish stronger scientific relationships between 
the fields, a common definition of the concept of energy is required. In crafting this definition, the relationships 
between mental models related to the concept of energy can be determined with the help of metaphors. Em- 
pathetic tendencies can also help determine the relationships between models and define metaphors. Future 
metaphor studies can investigate mental models related to the concept of energy within the context of prelimi- 
nary scientific information. It is necessary to identify students’ preliminary information regarding how and why a 
phenomenon takes place and reorganize the teaching-learning models for the science class in a way to address 
students’ misconceptions (Dekkers & Thijs, 1998; Osborne & Wittrock, 1983). Students’ preliminary knowledge about 
the concept of energy can be identified through metaphors. As metaphors serve as an interface between differ- 
ent pieces of information in the mental structuring of scientific knowledge, they can be used in forming mental 
maps related to the concept of energy. These maps can be used to correctly define the concept of energy as well 
as to eliminate confusion related to the properties of energy and energy as it is taught during class time. Mental 
models that facilitate the comprehension of the concept of energy can be used to produce a definition of energy 
that is common across different disciplines. 


Problem Situation 


Since metaphor and empathy are abstract concepts, their definitions vary from person to person and ac- 
cording to scientific fields. Definitions are made using similar characteristics of different concepts, phenomena, 
relationships, events, etc. Since similar characteristics vary from person to person, it is difficult to create a common 
definition for abstract concepts. People in scientific fields express the same concept as a metaphor using mental 
processes according to different definitions or specific characteristics. Therefore, metaphor sheds light on how the 
mental structure is formed. In addition, the use of metaphor will contribute to a common definition of abstract 
concepts among scientific fields. In this context, the concept of energy is expressed with different metaphors in 
different fields of science. These expressions will enable the establishment of connections between scientific fields, 
the elimination of deficiencies and errors, and the establishment of correct and effective relationships between 
fields. It is important to establish a correlation between concepts in the mental structuring of abstract concepts. 
Determining the correlation between energy metaphor and empathic tendency is important in terms of mental 
structuring. Learning is realized by establishing connections between mental structures. Knowing the connections 
between concepts will enable learning to take place more effectively. 

In this study, qualitative and quantitative relationships between the classification of energy metaphor ac- 
cording to traditional and scientific fields and the empathic tendency status and empathic tendency levels were 
examined. It was investigated whether there was a quantitative correlation between empathic tendency scores 
and traditional metaphor classes. A qualitative correlation between empathic tendency levels and the fields of sci- 
ence (categories) in which the energy metaphor is included was tried to be determined. The correlation between 
energy metaphor and empathy was determined with the following questions. 

1- What are the traditional classes and categories of the metaphors produced by pre-service mathematics 
and science teachers for the concept of energy? 

2- Whatare the empathic tendencies and empathic steps of pre-service mathematics and science teachers? 

3- Is there a significant difference between empathic tendency scores according to the traditional Energy 
Metaphor type? 


Research Methodology 


In the study, a semi-structured form and an empathic disposition scale were used to determine metaphors for 
the concept of energy. Mathematics (40) and science (71) pre-service teachers took part in the study. The concept 
of energy is very common in the fields of physics, chemistry and biology in science. Therefore, pre-service science 
teachers were included more in the study. 

The metaphors formed by the pre-service teachers for the concept of energy were categorized according to 
traditional and scientific fields. The empathic tendencies and empathic tendency steps of the pre-service teachers 
were determined. The correlation between the traditional metaphor classes and empathic tendency scores and 
the science fields (categories) in which the energy metaphor is included and the empathic tendency steps were 
determined. 
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The importance of the study is that concrete and abstract concepts are used as metaphors to define abstract 
concepts. The metaphors produced for the concept of energy contribute to the determination and definition of 
the mental structuring of the concept of energy. The fields of science use the concepts, events and objects in their 
own fields of science to define the concept of energy. This means that each field of science produces metaphors 
for the concept of energy according to its own mental structuring. It is to define the concept of energy according 
to the fields of science. By using the energy metaphor, a common definition of energy between the fields of sci- 
ence is provided and the connections between the fields of science are strengthened. It will also contribute to the 
definition of a common energy concept that can be used by science fields to determine learning situations. In 
addition, determining the connection between the abstract concept of empathic tendency and energy metaphor 
will contribute to the more comprehensive mental structuring and definition of the concept of energy. 

In the study process, candidates were given metaphors and examples of metaphors including the feature 
that connects the metaphor to the subject. While producing metaphors, it was emphasized that the metaphor 
alone would not make sense and that the feature linking the metaphor to the subject should be specified. A semi- 
structured form was created for the concept of energy and the pre-service teachers were asked to express their 
mental reflections about the concept of “energy” with another name using the form. The metaphors created for the 
concept of energy were obtained through content analysis. If the metaphor created for the concept of energy did 
not have energy-binding properties, it was considered invalid. The valid metaphors determined by the researchers 
were classified according to the traditional metaphors and science fields in Figure 1. The empathic tendency scale 
(Table 1) was used to determine the empathic tendency scores and empathic training stages of the teacher candi- 
dates. It was determined whether there was a quantitative correlation between traditional metaphor classes and 
empathic tendency scores. The fields of science (categories) in which the metaphors were created were compared 
with the empathic disposition levels. 

In this study, metaphors related to the concept of energy were classified according to the traditional metaphor 
classification in the “Dictionary of Linguistic Terms” (SIL, 2018). The International Summer Institute of Linguistics 
(SIL, 2018) made a classification based on the work of Lakoff and Johnson (1980), Beekman and Callow (1974) and 
Mish (1991). This classification is given in Figure 1: 


Figure 1 
Classification of Metaphors. SIL. (2018) 


Metaphors | 


| i 


Conventional Metaphors 1 [ Mixed Metaphors | [ New Metaphors. 


| Structural Metaphors | | 3 esis ony 
- ‘Container Metaphors 

| Substance Metaphors 

Lf Entity Metaphors 


Conventional metaphors: A conventional metaphor is commonly used in the colloquial language of a culture 
to give structure to part of the conceptual system of that culture. There are three different kinds of conventional 
metaphors: ontological, orientational, and structural. A structural metaphor is a conventional metaphor, whereby a 
concept is understood and expressed vis-a-vis another structured, sharply defined concept. Orientational metaphors 
involve orientations, such as space, place, inside, and outside, and are concerned with systematic relationships 
between emotions and motor-perceptual experiences. An ontological metaphor is a metaphor where an abstrac- 
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tion, such as an activity, emotion, or idea, is represented by something concrete, such as an object, substance, 
container, or person. There are three kinds of ontological metaphors: container, entity, and substance. Container 
metaphors are ontological metaphors in which some concepts are represented as having an inside and outside 
and capable of holding something else. Material metaphors are ontological metaphors in which an abstraction, 
suchas an event, activity, emotion, or idea, is represented as material. Entity metaphors are ontological metaphors 
in which an abstraction is represented as a concrete physical object (SIL, 2018). 

Mixed metaphors: Mixed metaphors are used to express the same concept, used in the same utterance, 
especially in the same sentence. In other words, it involves expressing a concept in a sentence using different 
metaphors. Mixed metaphors often, but not always, lead to a clash of concepts. 

New metaphors: New metaphors are not part of a culture’s conceptual system as reflected in its language. 
This is defined as allegory (SIL, 2018). Allegories are metaphors extendable as stories, especially used to make 
sense of and express aspects of fictional characters, actions, and concepts related to human existence (Kurt, 2019). 

The empathetic tendencies of pre-service teachers were measured using the empathic tendency scale devel- 
oped by Dokmen (1988) in Table 1. The scale is Likert type and consists of 20 question statements about empathic 
tendency. The lowest score that can be obtained from the scale is 20 and the highest score is 100. The highest score 
for each item is 5 and the lowest score is 1. 


Table 1 
The Empathic Tendency Scale (Dé6kmen, 2018) 
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1 | have many friends. () ) ) () () 
2 Sometimes | get teary-eyed while watching a movie. () ) ) () () 
3 [often feel lonely. () ) ) () () 
4 Those who talk to me about their problems feel relieved after our conversation. () ) ) () () 
5 The problems of others concern me as much as my problems. () ) ) () () 
6 | have difficulty communicating my feelings to others. () ) ) (3) () 
7 | find it strange when people cry while watching a movie () ) ) () () 
8 Sometimes, when I’m arguing with someone, my attention is more focused on my () () ) () (3) 

answer than on what the other person is saying. 

9  |ama very popular person in my social circle. () ) ) () () 
10 | feel relieved when movies have a happy ending. () ) ) () () 
11 | have difficulty communicating my thoughts to others. () ) ) () () 
12 — Most people are selfish. () ) ) () () 
13 ’'man angry person. () ) ) () () 
14 | usually trust people. () ) ) () () 
15 People don’t completely understand me. () ) ) () () 
16 I'ma sociable person. () ) ) () () 
17 Itmakes me feel better to open up to someone close to me about my problems. () ) ) () () 
18 lam generally satisfied with my life. () ) ) () () 
19  Myclose ones often tell me about their problems. () ) ) () () 
20‘ 'musually in a good mood. () ) ) () () 
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A high total score indicates a high level of empathetic tendency; a low score indicates a low empathetic 
tendency. Using the medians of each empathic tendency statement in the scale, the ranking of the empathetic 
tendency levels was determined. Mean empathic tendency scores vary between 1 and 5. According to the mean 
empathic scores, the empathic disposition behaviors of the pre-service mathematics and science teachers were 
ranked. According to this ranking, empathic disposition classes were determined. 

The metaphors about the concept of energy created by pre-service mathematics and science teachers were 
traditionally classified and categories were formed. Empathic tendency scores and average empathic scores of the 
candidates were calculated. Mann Witney-U test was used in the comparison of the traditional classifications of 
the metaphors formed by the candidates and their empathic tendency scores, since normality and homogeneity 
of variance were not ensured. 


Data Collection 


In the semi-structured form, sample metaphors were presented, and the pre-service teachers were asked to 
examine them. The pre-service teachers were asked to write a metaphor for and a property of the concept of energy 
in the blanks in the form. In the form, the first blank is for the source of the metaphor, and the second blank is for 
the property associated with the concept. Below is the template used by the pre-service teachers. 

Energy is ....... (source of the metaphor)....... Because......(the property that links the metaphor to the concept)... 

The metaphors produced in accordance with the template were classified traditionally in Figure 1 and Table 
2 and Table 3 of pre-service mathematics and science teachers were created. In addition, the fields of science in 
which the energy metaphors took place are shown in Figure 2 and Figure 3. 

The empathic disposition levels of the pre-service teachers were measured using the 20-item “empathetic 
disposition scale” (Table 1) developed by Dokmen (1988). Each question was answered on a Likert scale from 1 to 5, 
the lowest and the highest scores that can be obtained from an item, respectively. The 3rd, 6th, 7th, 8th, 11th, 12th, 
13th, and 15th items were reverse scored. Based on the scores obtained from the scale, the pre-service teachers’ 
empathetic tendency scale scores and the mean scores of the items were determined. 


Data Analysis 


The metaphors for the concept of energy and the features linking the metaphor to the source subject were 
obtained using a semi-structured form. The energy metaphor was created by passing the data through the fol- 
lowing stages (Saban, 2009). 

1- Coding and sorting: The data obtained through the semi-structured form were coded and sorted, and 
the metaphors developed by the participant teachers for the concept of energy and the associated 
property that links the metaphor to the subject were determined. 

2- Sorting and presenting valid metaphors in a table: The metaphors the pre-service teachers developed 
were classified under the respective types of conventional metaphors and tables were created accord- 
ingly. 

3- Categorization: The valid metaphors were categorized by their characteristics. 

4- Validity and reliability: Appropriate sample metaphors, instructions on what the metaphors should 
entail, the purpose of the study, and information about the concept of metaphors were laid out ina 
semi-structured form. The metaphors produced by the pre-service teachers about the concept of energy 
were examined by the researchers to check whether the metaphors and the associated properties were 
consistent, and the inconsistent ones were removed. 

5- Analysis of qualitative data: After the stages listed above, Table 2 and Table 3 were created, which 
contain the metaphors, the number of each metaphor, and the classification of the metaphors under 
types of conventional metaphors. In addition, the distribution of energy metaphors in Table 2 and Table 
3 according to science fields (categories) are given in Figure 1 and Figure 2. 

The data on empathic tendency consisted of the mean item scores and total scores of the pre-service teachers 
on the empathic tendency scale (Table 1). The mean and total scores of each item were found separately for 20 
question items. Means were found for each question in Table 1 and are presented in Table 4 and Table 5. The sum 
of the mean scores constituted the empathic tendency scores of pre-service mathematics and science teachers. 

The empathic disposition score corresponding to the traditional classification of the energy metaphor created 
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by each pre-service teacher was determined (Table 6 and Table 8). The averages of the empathic tendency scores 
of the candidates who produced a common energy metaphor were taken. The traditional ontology metaphor 1 
was represented by the structural metaphor 2 and the correlation between metaphor and empathic disposition 
was determined using the Mann-Whitney U test. The test results are listed in Table 7 and Table 9. 

The findings are presented under three sub-sections: the categorization of the metaphors and their classifica- 
tion under different types of conventional metaphors, pre-service teachers’ empathetic tendency levels, and the 
correlation between different types of metaphors and levels of empathetic tendency. 


Research Results 


The classification of the pre-service teachers’ metaphors for the concept of energy under different types of 
conventional metaphors and the categorization of the metaphors were made separately for mathematics and sci- 
ence pre-service teachers. Findings on the classification of the pre-service teachers’ metaphors for the concept of 
energy under different types of conventional metaphors and the categorization of the metaphors. Results related 
to the metaphors created for the concept of energy are presented separately for pre-service mathematics and sci- 
ence teachers. The metaphors developed and their classification under two traditional metaphor types are given 
in Table 2 and Table 3. Categories are created according to the metaphors in Table 2 and Table 3. 

The findings related to the empathic disposition levels of the participating pre-service mathematics and 
pre-service science teachers are given in Table 4 and Table 5. In the tables, empathic disposition scale items, mean 
scores for each item, and total scores are given. Total scores were obtained by summing the average score of each 
question item. 

Findings on the correlation between the classification of pre-service teachers’ metaphors about the concept 
of energy under traditional metaphor types and pre-service teachers’ empathic tendency levels. Table 6 and Table 
8 were created based on the data in Table 2 and Table 3, in which the participant pre-service teachers’ metaphors 
about the concept of energy were classified under different traditional metaphor types. Using these tables, the 
correlation between the metaphors classified as traditional and empathic tendency scores was determined using 
the SPSS software program. Since empathic tendency scores did not show normal distribution, Mann-Whitney U 
test was used, and the test results are given in Table 7 and Table 9. 


Traditional Classification of Energy Metaphors Produced by Pre-Service Mathematics Teachers. 
The pre-service mathematics teachers produced 40 metaphors for the concept of energy. The main concepts 
used by these pre-service teachers in creating metaphors were vitality and life (Table 2). These metaphors are clas- 


sified as structural (n = 27) and ontological (n = 13) (entity) metaphors. 


Table 2 
Traditional Classification of Energy Metaphors Generated by Pre-service Mathematics Teachers 


Metaphor foreneray N Kind of conventional Metaphor for N Kind of conventional 
metaphor energy metaphor 
Vitality 7 Structural Motivation 1 Structural 
Life 4 Structural A book 1 Ontological (Entity ) 
Food 2 Ontological (Entity ) Entrepreneurship 1 Structural 
Water 2 Ontological (Entity ) Aneed 1 Structural 
Movement 2 Structural Power 1 Structural 
Amischievous child 2 Ontological (Entity ) Aradiator 1 Ontological (Entity ) 
Happiness 2 Structural Breathing 1 Structural 
Acake 1 Ontological (Entity ) Nutrition 1 Ontological (Entity ) 
Achild 1 Ontological (Entity ) Playing a game 1 Structural 
An emotion 1 Structural Walking 1 Structural 
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Kind of conventional Metaphor for Kind of conventional 
Metaphor for energy N N 
metaphor energy metaphor 
A friend 1 Structural A daily activity 1 Structural 
Joy 1 Structural Astate of mind 1 Structural 
The sun 1 Ontological (Entity ) Atree 1 Ontological (Entity ) 
Total = 40 


5 Below is the categorization of the metaphors presented in Table 1, 


Energy as the perpetuity of life: The metaphors under this category are life (n=4), food (n = 2), water 
(n = 2), happiness (n = 2), cake (n = 1), child (n = 1), emotion (n = 1), friend (n = 1), joy (n = 1), motivation 
(n= 1), book (n= 1), entrepreneurship (n = 1), need (n = 1), breathing (n = 1), food (n= 1), playing a game 
(n= 1), daily activity (n = 1), and state of mind (n= 1). A total of 24 metaphors fell under this category, and 
in their metaphors, the said pre-service teachers used the concept of life the most. 


Energy as an aspect of the science of biology: Eight metaphors fell under this category, including 
vitality (n = 7) and tree (n= 1). 


Energy as an aspect of the science of physics: Six metaphors fell under this category, including move- 
ment (n = 2), a mischievous child (n = 2), power (n = 1), and walking (n= 1). 


Energy as an energy source: Two metaphors fell under this category, namely, sun (n = 1) and radiator 
(n= 1). 


Figure 2 
Distribution of Energy Metaphors Produced by Mathematics Teacher Candidates by Science Fields (categories). 


Energy as an energy source 


Energy as an aspect of science of physics 


Energy asan aspect of science of biology 


Energy as the perpetuity of life 


0% 10% 20% 30% 40% 50% 60% 70% 


Figure 2 shows the distribution of the metaphors produced by pre-service mathematics teachers for the 
concept of energy according to their fields of science. 


Traditional Classification of Energy Metaphors Generated by Pre-service Science Teachers 


The pre-service science teachers produced 71 metaphors for the concept of energy. The main concepts used 
by these pre-service teachers in creating metaphors were water, existence, money, mobility, food, and friends in 
generating metaphors for the concept of energy (Table 3). These metaphors are classified as structural (n = 40) and 
ontological (n = 31) (entity) metaphors. 
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Table 3 
Traditional Classification of Energy Metaphors Formed by Pre-service Science Teachers 


Metaphor for energy N Kind of conventional Metaphor for N Kind of conventional 
metaphor energy metaphor 
Water 6 Ontological (Entity ) An emotion 1 Structural 
Existence 6 Structural Reunion 1 Structural 
Money 6 Ontological (Entity ) A telephone dl Ontological (Entity ) 
Movement 5 Structural Dessert 1 Ontological (Entity ) 
Food 4 Ontological (Entity ) Open-air 1 Structura 
A friend 3 Structural Joy 1 Structura 
Vitality 2 Structural Walking 1 Structura 
Standing 2 Structural Aneed 1 Structura 
Happiness 2 Structural Power 1 Structura 
Work 2 Structural Effort 1 Structura 
Light 2 Ontological (Entity ) Motivation 1 Structura 
Doing work 2 Structural An atom 1 Ontological (Entity ) 
Nutrition 1 Ontological (Entity ) Awave 1 Ontological (Entity ) 
Life 1 Structural The Universe 1 Structura 
Electricity 1 Ontological (Entity ) Indestructible 1 Structura 
Fuel 1 Ontological (Entity ) Infinite 1 Structura 
A factory 1 Ontological (Entity ) Diddling 1 Structura 
Amachine 1 Ontological (Entity ) Human 1 Ontological (Entity ) 
The sun 1 Ontological (Entity ) Moving 1 Structura 
A battery 1 Ontological (Entity ) Transformation 1 Structura 
Matter 1 Ontological (Entity ) 
Total = 71 


Below is a categorization of the metaphors presented in Table 2. 


Energy as the perpetuity of life: The metaphors under this category are water (n = 6), existence (n = 6), 
money (n = 6), food (n = 4), friend (n = 3), happiness (n = 2), work (n = 2), nutrition (n = 1), life (n = 1), fac- 
tory (n= 1), machine (n = 1), emotion (n = 1), reunion (n = 1), telephone (n = 1), dessert (n = 1), open-air 
(n = 1), joy (n= 1), need (n = 1), effort (n = 1), and motivation (n = 1). A total of 42 metaphors fell under 
this category, and in their metaphors, the said pre-service teachers used the concepts of water, existence, 
and money the most. 


Energy as an aspect of the science of physics: A total of 16 metaphors fell under this category, includ- 
ing movement (n = 5), standing (n = 2), doing work (n = 2), walking (n = 1), power (n = 1), the universe 
(n= 1), indestructible (n = 1), infinite (n = 1), diddling (n= 1), and moving (n= 1). 


Energy as an energy source: A total of 16 metaphors fell under this category, including light (n = 2), 
electricity (n = 1), fuel (n = 1), sun (n = 1), battery (n = 1), and wave (n = 1). 


Energy as an aspect of the science of biology: Three metaphors fell under this category, including 
vitality (n = 2) and human (n= 1). 


Energy as an aspect of the science of chemistry and physics: Three metaphors fell under this category, 
namely matter (n = 1), atom (n= 1), and transformation (n= 1). 
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Figure 3 
Distribution of Energy Metaphors Produced by Pre-service Science Teachers According to Science Fields (category) 
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Figure 3 shows the distribution of the metaphors produced by pre-service science teachers for the concept 
of energy according to their fields of science. 

The categorical distributions of energy metaphors created by pre-service mathematics and science teachers 
are similar. Unlike science teachers, mathematics teachers predominantly produced metaphors in the category of 
chemistry and physics. While 41% of the metaphors produced by science teachers were related to science (Figure 
3), this rate was 40% for mathematics teachers (Figure 2). Pre-service mathematics and science teachers produced 
the most metaphors in the category of the perpetuity of life (60% and 59%, respectively). 


Empathic Tendency Levels of Pre-service Mathematics Teachers 


The responses of the pre-service mathematics teachers to the statements in the empathetic tendency scale 
were ranked from the one with the highest mean score to the one with the lowest. The closer the mean score to 5 
the higher the level of empathetic tendency. 

As seen in Table 4, the total of the mean scores pre-service mathematics teachers obtained from the items 
of the empathic tendency scale was found to be 69.57. The participant pre-service mathematics teachers were 
found to exhibit a high level of empathetic tendency. The items whose mean scores are between 3.72 and 4.17 
showcase a high level of empathetic tendency. These items are related to helping others and solving others’ prob- 
lems, which are, according to the stages of empathy proposed by Dékmen (1988), associated with the you stage. 
Individuals who respond at this stage show behaviors such as supporting others, addressing problems, repetition, 
and understanding deep feelings. 

The items whose mean scores are between 3.15 and 3.70 showcase an above-moderate level of empathetic 
tendency (Table 4). Items that point to an above-moderate level of empathetic tendency are associated with the 
| stage of Dokmen’s (1988) stages of empathetic tendency, meaning that individuals practice empathy with a 
self-centric approach. Individuals who respond at this stage make empathetic remarks such as “I feel” and “I think”. 
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Table 4 
The Mean Scores of The Items of Empathic Tendencies of Pre-service Mathematics Teachers 


Empathie tendency Empathetic tendency statements (n = 40) Average empathelle 
questions disposition scores 
7 | find it strange when people cry while watching a movie 4.17 
10 | feel relieved when movies have a happy ending. 3.91 
5 The problems of others concern me as much as my problems. 3.9 
4 Those who talk to me about their problems feel relieved after our conversation. 3.85 
19 My close ones often tell me about their problems. 3.72 
6 have difficulty communicating my feelings to others. 3.7 
3 often feel lonely. 3.7 
17 t makes me feel better to open up to someone close to me about my problems. 3.67 
9 am a very popular person in my social circle. 3.65 
20 ’m usually in a good mood. 3.62 
18 am generally satisfied with my life. 3.42 
3 Sometimes, when I’m arguing with someone, my attention is more focused on 33 
my answer than on what the other person is saying. 
11 have difficulty communicating my thoughts to others. 3.3 
13 ’m an angry person. 3.25 
16 ’ma sociable person. 3.22 
2 Sometimes | get teary-eyed while watching a movie. 3.2 
15 People don’t completely understand me. 3.17 
12 Most people are selfish. 3.05 
1 | have many friends. 2.95 
14 | usually trust people. 2.82 
Total 69.57 


The items whose mean scores are between 2.82 and 3.05 showcase a moderate level of empathetic tendency 
(Table 4), which is associated with making friends, trusting, and selfishness. This level is associated with the they 
stage proposed by Dékmen (1988). Individuals who respond at this stage are concerned with what society (they) 
think or feel about an individual's problems. The most prevalent stage at which pre-service mathematics teachers 
respond is the you stage, followed by the | stage and the they stage. 


Empathic Tendency Levels of Pre-service Science Pre-Service Teachers 


The responses of the pre-service science teachers to the statements in the empathetic tendency scale were 
ranked from the one with the highest mean score to the one with the lowest. 

As seen in Table 5, the total mean score pre-service science teachers obtained for the items of the empathic 
tendency scale was found to be 68.86. The participant pre-service science teachers were found to exhibit a high 
level of empathetic tendency. The items whose mean scores are between 3.91 and 4.02 showcase a high level of 
empathetic tendency. These items are related to helping and supporting others and solving others’ problems, which 
are, according to the stages of empathy proposed by Dékmen (1988), associated with the you stage. 
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Table 5 
The Mean Scores of The Items of Empathic Tendencies of Pre-service Science Teachers 


Empathic fehuency Empathetic tendency statements (n = 71) Average empatnetie 
questions disposition scores 
5 The problems of others concern me as much as my problems. 4.02 
4 Those who talk to me about their problems feel relieved after our conversation. 4.01 
10 feel relieved when movies have a happy ending. 3.97 
7 find it strange when people cry while watching a movie. 3.91 
3 often feel lonely. 3.85 
9 am a very popular person in my social circle. 3.85 
19 My close ones often tell me about their problems. 3.83 
6 have difficulty communicating my feelings to others. 3.73 
20 ’m usually in a good mood. 3.7 
16 ‘ma sociable person. 3.61 
18 am generally satisfied with my life. 3.4 
2 Sometimes | get teary-eyed while watching a movie. 3.35 
1 have many friends. 3.25 
11 have difficulty communicating my thoughts to others. 3.22 
17 it makes me feel better to open up to someone close to me about my problems. 3.19 
13 ’m an angry person. 3.12 
15 People don’t completely understand me. 3.05 
8 Sometimes, when I’m arguing with someone, my attention is more focused on my answer 278 
than on what the other person is saying. 
12 Most people are selfish. 2.64 
14 | usually trust people. 2,38 
Total 68.86 


The items whose mean scores are between 3.12 and 3.85 showcase an above-moderate level of empathetic 
tendency. Items that point to an above-moderate level of empathetic tendency are associated with the | stage of 
Dékmen’s (1988) stages of empathetic tendency, meaning that individuals practice empathy with a self-centric 
approach. The items whose mean scores are between 2.38 and 3.05 showcase a moderate level of empathetic 
tendency, which is associated with a lack of understanding of the other person, distrust, and selfishness This level 
is related to the tens level suggested by Dékmen (1988). The most common stage to which pre-service science 
teachers responded was the you stage, followed by the | stage and the they stage. 


Relationship Between Traditional Energy Metaphors Constructed by 
Pre-service Mathematics Teachers and Their Empathic Tendencies 


Table 6 shows the mean scores of the traditional energy metaphors and empathic tendencies of pre-service 
mathematics teachers. Ontological metaphors were coded as 1 and structural metaphors as 2. Since the mean 
scores of empathic tendencies did not show normal distribution, the correlation between metaphor and empathic 
tendency was found using Mann-Whitney U test. 
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Table 6 
Mathematics Teachers’ Mean Empathic Tendency Scale Scores and the Types of Conventional Metaphors They Produced 


Empathic Kind of Empathic Kind of 
Energy P Energy i 
N tendency conventional N tendency conventional 
metaphor metaphor 
score metaphor score metaphor 
Vitality 7 73.28 Structural Motivation 1 46 Structural 
Life 4 78 Structural A book 1 80 Ontological(Entity) 
Food 2 62 Ontological(Entity) Entrepreneurship 1 46 Structural 
Water 2 69 Ontological(Entity) Aneed 1 78 Structural 
Movement 2 69 Structural Power 1 56 Structural 
A mischievous child 2 73.5 Ontological(Entity) A radiator 1 75 Ontological(Entity) 
Happiness 2 61 Structural Breathing 1 62 Structural 
Acake 1 78 Ontological(Entity) Nutrition 1 75 Ontological(Entity) 
Achild 1 63 Ontological(Entity) Playing a game 1 80 Structural 
An emotion 1 75 Structural Walking 1 56 Structural 
A friend 1 73 Structural A daily activity 1 78 Structural 
Joy 1 83 Structural Astate of mind 1 62 Structural 
The sun 1 75 Ontological(Entity) A tree 1 68 Ontological(Entity) 


The results of the Mann-Whitney U test to compare the empathic tendency scores according to the traditional 
energy metaphor types created by the pre-service mathematics teachers are shown in Table 7. In this table, no 
statistically significant relationship was found between the scores of the empathic tendency scale according to 
the traditional metaphor types according to the Mann-Whitney U test results (U = 174; p > .965). 


Table 7 
Comparison of Pre-service Mathematics Teachers’ Empathic Tendency Scores According to The Traditional Metaphor Types 


Traditional metaphor types N MR SR U p 
Ontological (entity) 13 20.86 265 
174 .965 
Structural 27 20.56 555 


Not: Mean Ranks: MR and Sun of Ranks: SR 


Relationship between Traditional Energy Metaphors Constructed by 
Pre-service Science Teachers and Their Empathic Tendencies 


Table 8 shows the mean scores of pre-service science teachers’ traditional energy metaphors and empathic 
tendencies. Ontological metaphors were coded as one and structural metaphors as 2. Since the mean empathic 
disposition scores were not normally distributed, the comparison of empathic tendency scores according to tra- 
ditional metaphor types, according to Mann-Whitney U test. 
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Science Teachers’ Mean Empathic Tendency Scale Scores and The Types of Conventional Metaphors They Produced 
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Table 9 shows the results of the Mann-Whitney U test, which was conducted to compare the scores of em- 
pathic tendencies according to traditional energy metaphors created by pre-service science teachers. According to 
this table, when the scores of the empathic disposition scale were compared with the traditional metaphor types 
produced by the pre-service teachers, there was no statistically significant difference on the empathic disposition 
scale score of the metaphor generation type (U = 527.5; p > .282). 


Table 9 


Comparison of Pre-service Science Teachers’ Empathic Tendency Scores According to the Traditional Metaphor Types 


Traditional metaphor types 


Ontological (entity) 


Structural 


N 


31 


40 


Note: Mean Ranks: MR and Sun of Ranks: SR 
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Discussion 


Mathematics and science teachers produced metaphors for the concept of energy by using abstract and 
concrete concepts. The metaphors produced were in the structural and ontological (entity) metaphor class in the 
traditional classification. The metaphors produced mostly consisted of structural and abstract concepts. Concrete 
objects and concepts were used as ontological metaphors. 


Traditional Classification and Categorization of the Metaphors Produced by 
Pre-service Teachers for the Concept of Energy 


The metaphors produced by pre-service mathematics teachers for the concept of energy were distributed 
as 27 structural and 13 ontological (being) metaphors according to the traditional metaphor classification. Pre- 
service science teachers distributed 40 structural and 31 ontological metaphors. While producing metaphors for 
abstract concepts, pre-service mathematics and science teachers mainly used structural metaphors. Pre-service 
mathematics teachers produced more structural metaphors than pre-service science teachers did. 

Pre-service mathematics teachers predominantly expressed the abstract concept of energy using abstract 
concepts. These concepts, such as vitality, life, and happiness, are used in daily life. Lakoff and Johnson (2003) stated 
that structural metaphors have an important place in describing daily events. While pre-service mathematics teach- 
ers produced metaphors for the concept of energy, they used concepts such as interaction, entrepreneurship, friend, 
need, play, playfulness, and mood. Inam (2008) stated that structural metaphors are frequently used in daily life and 
communication. Pre-service mathematics teachers produced 13 ontological (being) metaphors by abstracting the 
abstract concept of energy. These metaphors are oriented toward daily needs and comprise concepts such as food, 
water, cake, and book. Ontological metaphors consist of concepts used to concretize abstract concepts (Akin, 2015). 

The metaphors produced by pre-service mathematics teachers for the concept of energy were categorized 
as 60% continuity of life, 20% biological science, 15% physical science, and 5% energy source (graph 1). Pre-service 
teachers produced metaphors about the sustainability of life rather than lessons. Of the metaphors produced, 
40% were related to lessons. 

While producing metaphors for the concept of energy, pre-service science teachers predominantly used the 
structural metaphor (n=40). Structural metaphors are composed of abstract concepts oriented toward daily life, 
suchas life, movement, friends, vitality, happiness, and living. Because energy is an abstract concept and there are 
differences between scientific and everyday energy perceptions, metaphors related to everyday concepts were 
predominantly produced. This emphasizes that students use terms related to the world they live in when trying to 
explain events or situations in science (YUrumezodlu et al., 2009). Pre-service science teachers produced 31 onto- 
logical (being) metaphors by abstracting the abstract concept of energy. These metaphors generally consisted of 
concrete concepts related to daily needs, such as water, money, food, food, sun, telephone, and dessert. Candidates 
produced a low number of metaphors for science, such as doing business, electricity, fuel, battery, matter, humans, 
atoms, and waves. Pre-service teachers used abstract concepts when expressing the abstract concept of energy. 
This is because the complex and abstract aspects of metaphors can be understood in more concrete, familiar, and 
easily imagined terms (Lancor, 2015; Refaie, 2001; Semino, 2008). According to conceptual theory, since conceptual 
systems are metaphorical, abstract concepts can be understood using concrete metaphors (Lakoff & Johnson, 2003). 
The human conceptual system is metaphorical, and it has been stated that abstract concepts can be understood 
through concrete concept metaphors (Lakoff & Johnson, 1980, 2003). 

The metaphors produced by pre-service science teachers for the concept of energy were 59% continuity of 
life, 23% physical science, 10% energy source, 4% biological science and 4% chemistry. The metaphors created by the 
candidates for science courses constitute 41% of the total metaphors. The number of metaphorical expressions 
related to the concept of energy in the field of science was low. In a study, it was stated that students had a lim- 
ited conceptual understanding of energy related to science lessons (Opitz et al., 2019). In general, the metaphors 
produced consisted of concrete and abstract phenomena necessary for human life. 


Empathic Tendencies of Pre-service Mathematics and Science Teachers 


The empathy tendency score of pre-service mathematics teachers was 69.57 and 68.86 for pre-service science 
teachers. According to the scores of the candidates, their empathetic tendencies can be said to be at a high level. 
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Although the ranking of the expressions of empathic tendency of pre-service mathematics and pre-service science 
teachers according to the mean score is different, the same empathic tendency expressions are in the first place. 
The expressions of empathic disposition of pre-service mathematics and pre-service science teachers with the 
highest mean were expressions such as supporting the other person, addressing their problems, and understanding 
their feelings. The pre-service teachers experienced the emotions, thoughts, and perspectives of the other person 
from their point of view. Empathy is about one being permeated by the other's self as though one were taking 
part in the other’s emotional state and experiencing the other's perspective and life (Zahavi & Overgaard, 2012). 
The pre-service teachers felt the other person's problem and emotion as their own and showed an empathetic 
tendency in this direction. These tendencies correspond to the you stage according to Dokmeni’s (1998) classifica- 
tion of empathetic tendencies. 

The expressions of empathic disposition that had the second-ranked mean for pre-service mathematics and 
science teachers were the expressions of empathetic disposition that included their feelings and thoughts. Can- 
didates preferred empathetic expressions that emphasized their feelings and thoughts rather than those of the 
other person. Expressions of own feelings and thoughts correspond to the/ step in the classification of empathetic 
tendencies. Pre-service mathematics and science teachers preferred empathetic disposition expressions such as 
what society thinks and feels in the face of their problems. These were empathetic expressions such as selfishness, 
distrust, and lack of understanding. These statements correspond to the tens level in the classification of empathic 
tendencies. 


The Correlation between Traditional Classes of Metaphors Related to the Concept of 
Energy and Empathic tendencies 


The correlation between the traditional metaphors created by pre-service mathematics and science teach- 
ers and their empathic tendency scores was found using Mann-Witney U test. It was determined that there was 
no statistically significant difference between the empathic tendencies of pre-service mathematics and science 
teachers and traditional metaphor classes. The empathetic dispositions of pre-service mathematics and pre-service 
science teachers were found to have the highest mean at the sen step and high empathetic disposition in the 
study. In the traditional metaphor classification of the pre-service teachers, structural metaphors were the most 
common. In addition, more than half of the total energy metaphors produced were in the category of continuity 
of life. If an inference is made from these, it can be that the candidates have a high level of empathetic tendencies, 
that the class of empathetic tendencies is the sen level, and that they mainly produced structural metaphors and 
categorically produced concrete and abstract metaphors for the continuity of life. 


Conclusions and Implications 


Abstract concepts such as energy and empathic disposition are difficult to define. The correct definition of 
abstract concepts contributes to increasing learning effectiveness and connections between disciplines. In order 
to define abstract concepts, common features of other concepts are used, and this is done through mental pro- 
cesses. Mental processes vary from person to person. It is important to use abstract concepts such as metaphor 
and empathic tendency to meet this change at a common point, which will contribute to determining mental 
structuring. In this study, the results of the 3 questions regarding the concepts of energy metaphor and empathic 
tendency are given in the following paragraphs. 

Mathematics and science teachers produced metaphors for the concept of energy by using abstract and 
concrete concepts. The metaphors produced were classified as structural and ontological (being) metaphors in 
the traditional classification. The metaphors produced by the pre-service teachers are structural and consist of 
abstract concepts. Concrete objects and concepts were used as ontological metaphors. 

Approximately 60% of the metaphors produced by pre-service mathematics and science teachers for the 
concept of energy were in the category of continuity of life. Approximately 40% of the metaphors produced were 
in the science categories. Biology is the prominent field focused on by pre-service mathematics teachers, while 
physics is the prominent field for pre-service science teachers. In generating metaphors for courses, pre-service 
mathematics teachers produced metaphors for biology science, while pre-service science teachers produced 
metaphors for physics science. 

Pre-service mathematics and pre-service science teachers have an above-average (high-level) empathic disposi- 
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tion and have the sen level as the empathic disposition level. Pre-service teachers feel the other person’s problems 
and emotions as their own. Pre-service teachers look at events and situations from the other person’s point of view. 

No quantitatively significant difference was found between traditional metaphor types and empathic tenden- 
cies. It can be inferred from this that while the pre-service teachers’ empathic tendencies were high and the “you” 
level was prominent while producing metaphors for the concept of energy, they produced structural metaphors 
from the concepts related to the sustainability of life the most. 


Recommendations 


Metaphors and empathy are abstract concepts and reflections of mental models. A contribution can be made 
to the determination of mental structuring by determining the models in empathy and metaphor formation. 
Metaphors can be used as learning outcomes in determining mental models and thoughts about the subject in 
education and training processes. As empathy and metaphors are reflections of mental models, they can be used 
in restructuring the educational process and learning outcomes. In addition, expressions of an empathetic disposi- 
tion can be used in teacher-student interactions during the lesson. The concept of energy is used in different fields 
of science. Using metaphors to establish connections between branches of science, the connections between 
branches of science can be extended. 
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DETERMINING PRE-SERVICE 
SCIENCE TEACHERS’ 
UNDERSTANDING ABOUT STEM 
EDUCATION 


Gonca Kececi 


Introduction 


Today, as we are getting closer to completing the first quarter of the 
21st century, the constant change in technology has required countries to 
adapt themselves to these changes. With every developing technology, 
new business areas emerge, creating the need for competent employment 
in these business areas. These obligations and needs have made it essential 
to raise future generations as individuals who can follow the ever-changing 
technologies and adapt. It has been noted that students who are expected 
to struggle with the ever-changing world problems with the integration of 
engineering in K-12 education will gain the advantage of contributing to 
their teamwork, communication skills and problem-solving skills (Brophy 
et al., 2008). At this point, STEM education, which aims to integrate science, 
technology, engineering and mathematics disciplines, has come to the 
fore with an interdisciplinary perspective. It has been suggested to create 
student-centered environments by integrating engineering with science 
standards (National Research Council [NRC], 2012). In order to popularize 
the approach led by the United States of America, recommendations for cur- 
riculum development in STEM fields have been made in published national 
reports (National Academy of Sciences, National Academy of Engineering, 
& Institute of Medicine of the National Academies, 2007). Calls have been 
made to encourage the applications of the STEM approach in K-12 educa- 
tion, to direct students to careers in STEM fields, and to increase their success 
in these fields (NRC, 2011). The Next Generation Science Standards, which 
encourage engineering and scientific applications rather than rote teaching, 
have been adopted by many states in America (NRC, 2013). Various reports 
and research, that have adopted the STEM approach in other countries, 
have been published one after another (Bureau of Labor Statistics, 2020; 
European Commission, 2015; EU STEM Coalition, 2015; Ministry of Educa- 
tion & Research, 2010; Ministry of Education [MNE], 2016; SINTEF, 2011; The 
Organization for Economic Co-operation and Development [OECD], 2008; 
OECD, 2013; Turkish Industry and Business Association [TUSIAD], 2017). These 
reports, whose common point is to provide learning environments suitable 
for STEM understanding, are not easy to implement. Despite recommen- 
dations to use the STEM approach, it is unclear exactly what it represents 
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understanding of the pre-service science 
teachers was tried to be determined 
before the theoretical STEM education, 
after the theoretical education and after 
the STEM application. The study group 

of the research consisted of a total of 66 
pre-service teachers. Content analysis 
results of the interviews were carried out in 
three stages. It was found that there was 
no single STEM definition that pre-service 
science teachers agreed on. The training 
provided increased the STEM competency 
levels of pre-service science teachers. 
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(Herschbach, 2011). It is seen that STEM is interpreted differently. There are different definitions such as; STEM as an 
integrated learning and teaching approach that requires making connections between disciplines (Karatas, 2017); 
as a simple term replacing mathematics or science (Sanders, 2009); accepting the argument that the combining 
of two disciplines is enough (Kelley & Knowles, 2016). According to Honey et al. (2014), in STEM applications, one 
discipline is used more dominantly than the others, while the others are used as supportive. In STEM discipline 
integration, disciplines can also be used as interdisciplinary, transdisciplinary or multidisciplinary (Vasquez et al., 
2013). Along with the different definitions of STEM education, another challenge awaiting practitioners is the use 
of different pedagogy practices (Dare et al., 2019). Problem-based STEM (Breiner et al., 2012; El Sayary et al., 2015; 
Dischino et al., 2011), project-based STEM (Capraro & Jones, 2013; Capraro & Slough, 2013; Han et al., 2015; Sahin, 
2013), engineering design-based STEM (Billiar et al., 2014; Capobianco & Rupp, 2014; Guzey et al., 2016), STEM with 
5E model (Bybee, 2019; Sanjaya, 2018) have been suggested to be used in the STEM approach. The STEM literacy 
levels of the teachers who will interpret the STEM approach, where there is such a diversity, are very important. 
Teachers have a great role to play in the trainings to be given for the STEM approach. It is necessary to develop 
teachers’ understanding of integrating STEM disciplines in addition to their knowledge of the content, pedagogy 
and technology (Honey et al., 2014). Although teachers are expected to integrate STEM disciplines, they face a 
lack of information on how to do so (Dare et al., 2019; Ejiwale, 2013). There is more than one application for the 
integration of STEM disciplines and there is no single truth that teachers will use (Bybee, 2013; Johnson, 2012; 
Ring et al., 2017). This status makes it hard for teachers who want to include STEM education in their classrooms. 
Before applying STEM education, teachers need to understand, make sense of, and determine the appropriate 
model for their own understanding (Ring et al., 2017). In cases where teachers are not given the opportunity to 
think about their own STEM models and appropriate environments are not provided. It can be difficult for them 
to take an interdisciplinary perspective and understand the concept of integrated STEM correctly and can cause 
contradictions in their beliefs (Moore et al., 2014). 


Research Problem 


The STEM approach has also been received with interest in Turkiye. In the report arranged by the TUSIAD 
(2014), itis underlined that countries that want to improve economy and technology will need persons who have 
studied STEM disciplines. Afterwards, in the STEM education report issued by the Ministry of National Education 
(2016), it was pointed out that individuals with 21st-century skills would also direct the forthcoming of countries. 
It keynoted the importance of STEM education in transforming theoretical knowledge into product, practice, and 
innovative inventions. With the influence of these reports, studies on the STEM approach have gradually increased 
since 2014 (Akaygun & Aslan-Tutak, 2016; Akgunduz et al., 2015; Baran et al., 2016; Corlu, 2014; Karahan et al., 
2015; Kececi et al., 2017; Marulcu & Hobek, 2014). With the applied science unit of the science course curriculum 
in 2017, the STEM approach, which has been previously given a start by private schools and institutions, took 
its place in the program. In the science curriculum for secondary schools organized in 2018, the applied science 
unit was abolished, and students were believed to assign a demand or problem from real life for the subjects in 
the units and make science, engineering and entrepreneurship practices (MNE, 2018). At this point, the biggest 
problem is that science teachers are caught unprepared for this situation and have difficulties in applying it 
(Blackley & Howell, 2015; Corlu et al., 2014). The surprise of teachers who have not been educated in any training 
on the STEM approach during their undergraduate education can be considered natural. Teachers who use STEM 
activities with their students during the education and training process should be competent in STEM education. 
How teachers comprehend STEM understanding, whether they embrace it or not, whether they see themselves 
as sufficient or not will directly affect the success of the program. However, teachers’ studies on how teachers 
perceive STEM education are limited compared to STEM definition studies (Dare et al., 2019). Although pre- 
service teachers participated in STEM practices, it was found that they were not confident enough in describing 
STEM education (Bartels et al., 2019). Alan et al. (2019), in their study that aimed to support pre-service science 
teachers’ integrated teaching knowledge through STEM applications, stated that pre-service teachers believed 
in the necessity of STEM education, but thought that discipline integration was not easy. When teachers’ STEM 
awareness is increased before they start their profession, they are given the opportunity to establish their own 
understanding of STEM before the application. In this regard, this study aimed to determine the understanding 
of pre-service science teachers about STEM education. 
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Research Focus 


The lack of a single accepted definition of STEM education, which is globally accepted and tried to be inte- 
grated into curricula, is an important problem for teachers and pre-service teachers who are STEM practitioners. 
However, the goal of effective workforce and career planning in the STEM field, which is desired to be achieved 
with STEM education, is common. Using STEM for equipping future generations with skills such as communication, 
cooperation, entrepreneurship, critical thinking and problem-solving required by the 21st century is also a com- 
mon goal of many countries. In this respect, knowing the STEM understanding of pre-service teachers who have 
not started the profession actively gives education policymakers the chance to make the necessary interventions. 
In the research, STEM education was determined as a phenomenon, and it was tried to determine how pre-service 
science teachers perceived STEM education, whether they adopted it, whether they saw themselves as sufficient, 
and the environment and situations that affected STEM experiences. In the research, pre-service science teachers 
first participated in the STEM theoretical training, which lasted for six weeks. The pre-service teachers, who were 
divided into groups after the theoretical training, prepared a STEM activity that they could implement with their 
students in the future according to a STEM understanding they chose, and a lesson plan on how to implement 
the activity. Pre-service teachers presented their activities and the STEM approach they adopted to their peers. 
Pre-service teachers in the role of listeners evaluated the activities of their peers in their individual diaries. 


Research Aim and Research Questions 


This research aimed to determine the understanding of STEM about STEM during the 14-week theoretical 
and applied STEM education processes of pre-service science teachers. In this research study, responses to the 
following questions were explored. 

1. What is the STEM understanding of pre-service science teachers before theoretical education and 
before and after applied education? 
2. Which STEM learning and teaching models do pre-service science teachers prefer to use? 


Research Methodology 
General Background 


A phenomenological research design was used in the research. The goal of research using phenomenological 
design is to search for various ways people use to comprehend, comment on, or make sense of a certain phenom- 
enon or a certain aspect of reality. With this search, meanings are revealed on facts and these meanings are clas- 
sified according to categories (Cepni, 2010). In phenomenological research, it is tried to understand and describe 
how people perceive these experiences by examining their daily experiences. In this paper, STEM education was 
determined as a phenomenon and pre-service science teachers’ understanding of STEM was researched. Analysis 
of personal text, focus meetings, conversations with participants, participant observation, action research and 
interviews can be used in phenomenological designs (Delve & Limpaecher, 2022). In this research, semi-structured 
interviews were used for collecting data. All pre-service science teachers participated on a voluntary basis. Interviews 
were carried out three times: before theoretical education, and before and after applied education. 


Participants 


The study, in which 49 female and 17 male pre-service teachers participated, was carried out at a state univer- 
sity in eastern Turkiye. The research was carried out within the content of the “Special Teaching Methods II course’, 
in a 14-week period in the 2018-2019 education term. The study group was settled by the purposeful sampling 
method. The researcher can take as some sample individuals who he believes reflect the generality and fit the 
characteristics he has determined, depending on his own judgment in this method (Ural & Kili¢, 2011). All of the 
pre-service teachers participated in STEM education applications, in which models related to renewable energy 
were designed using educational Lego sets before the research. In addition, all of the pre-service teachers attend 
the school experience course. Criteria for the study group of the research were preferred because it is thought 
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that more realistic comments will be received in terms of the fact that the pre-service teachers have completed 
the basic courses, they have STEM awareness, and they see the practices of the teaching profession in the real 
environment during the school experience lesson. While coding the pre-service teachers, the ordinal numbers of 
the students who took the course and the group numbers during the STEM applications were coded to be written 
one after the other. For example, “PT37” was written for the 37th pre-service teacher in the class list, then the group 
number was added as“G11” and coded as“PT37G11” The research process was explained to the pre-service teach- 
ers, and it was stated that there was a voluntary basis for the collection of data within the scope of the research. 
All of the participating pre-service teachers voluntarily agreed to take part in the research. However, there were 
14 pre-service teachers (PT3G10, PT5G7, PT7G9, PT9G10, PT15G10, PT31G5, PT40G2, PT41G9, PT45G11, PT47G9, 
PT57G11, PT62G11, PT63G2, PT65G5) who could not be reached for the interviews in the last stage of the study 
because it was the end of the semester, and the pre-service teachers went to other provinces due to the holiday 
period. However, since the data were collected at the beginning and in the middle of the application, the pre- 
service teachers were not excluded from the study group. 


Instrument and Procedures 


Pre-service teachers attended the interactive training in the 2018-2019 academic year in a 14-week period, in 
which theoretical information about STEM theory and applications was given. STEM definitions, history of STEM, 
national and international reports on STEM, learning-teaching models used in STEM education and STEM applica- 
tion examples were shared with pre-service teachers during the six-week period. After the theoretical knowledge, 
the pre-service teachers were released to form their groups for the application phase. It has been suggested that 
they form groups of 5-6 people and that the number of male pre-service teachers (25%) is less than the number 
of female pre-service teachers (75%) so that they should be homogeneously distributed among the groups. 
However, at the end of the given week, the pre-service teachers formed 11 groups that were quite different from 
what was expected. The 66. pre-service teachers did not want to join any group and wanted to carry out their own 
study. The fact that the groups were formed in different numbers and features suggested that it may be due to the 
differentiation of the communication and cooperation skills of the pre-service teachers, and the groups formed 
were not intervened in order to monitor how the process would continue. The experiences of pre-service teachers, 
who are expected to do STEM applications with their students in their future classes, while implementing their 
own STEM projects in this process are thought to be very important. In addition, it is considered as an important 
gain that pre-service teachers have the chance to observe and evaluate different projects in the other groups. The 
groups constructed an instance lesson plan in line with the STEM understanding they adopted, carried out STEM 
applications, presented their projects to their peers, and defended the dimensions of STEM disciplines. Pre-service 
teachers appreciated and discussed the STEM practices presented by their peers in terms of whether they should be 
integrated into disciplines or not, in terms of the method used. The presentations of the pre-service teachers were 
completed with self, peer and teacher evaluations. Presentations were recorded in order to understand how the 
pre-service teachers’ understanding of STEM was shaped and how the learned theoretical knowledge was reflected 
in practice. Pre-service teachers carried out 11 group and one individual STEM activities. In the implementation 
phase, where the pre-service teachers were expected to do a STEM project in which they integrated the disciplines 
with their groupmates, all groups, except Group 10 and PT66G12 (who worked individually) preferred to use sen- 
sor sets to integrate technology and content-based engineering (Blackley & Howell, 2015; Moore & Smith, 2014) 
have included the gains of software engineering by coding in accordance with the integration. The number of 
people in the groups formed and information on the teaching and learning models the groups adopted in STEM 
applications are in Table 1. 


Table 1 
Characteristics of the Groups Created 


Group Female Male Adopted STEM models 
1 PT16, PT19, PT33, PT36, PT38 PT1 Project Based STEM 
2 PT24, PT29, PT40, PT58, PT63 PT49 Design Based STEM 
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Group Female Male Adopted STEM models 

3 PT12, PT32, PT34, PT56, PT59 PT22 Problem-Based STEM 

4 PT6, PT18, PT25, PT50, PT52 PT30 Problem-Based STEM 

5 PT31, PT39, PT54, PT65 PT4, PT8 Problem-Based STEM 

6 PT10, PT14, PT28, PT37, PT60, PT53 PT2 Problem-Based STEM 

7 PT5, PT11, PT48, PT55, PT61 PT42 Problem-Based STEM 

8 PT23, PT27, PT44 - Project Based STEM 

9 PT13, PT17, PT35, PT41, PT46, PT47, PT64 PT7 Problem-Based STEM 

10 PT15, PT20, PT21, PT43 PT3, PT9 Problem-Based STEM 

11 - PT26, PT45, PT51, PT57, PT62 Project Based STEM 

12 PT66 - Project Based STEM 


When Table 1 is examined, it is shown that seven of the groups formed by the pre-service teachers consist of six 
people, one group consists of seven and one group consists of eight people. It is shown that all members of Group 
11 are male, and all members of Group 8 are female pre-service teachers. On the other hand, it is shown that a pre- 
service teacher prefers individual work rather than collaborative group work. It is shown that four groups adopted 
project-based STEM, seven groups adopted problem-based STEM, and one group adopted an engineering-based 
STEM approach in their STEM activities. 

The data of the study were gathered through semi-structured interviews participated on a voluntary basis of 
pre-service teachers. Interviews were carried out three times: before theoretical education, and before and after 
applied education. Interviews lasted between 10-30 minutes. The interviews of the pre-service teachers were 
recorded using a voice recorder with the authorization. 


Data Analysis 


Phenomenological data analysis stages, textural descriptions in which data are directly listed, important ex- 
planations and quotations are made to understand how the phenomenon is experienced, keywords are specified, 
expressions are specified within themes; structural descriptions, in which a description of the setting or context that 
influences the way participants experience the phenomenon; and the “essence” of the phenomenon is conveyed 
as composite descriptions (Moustakas, 1994). In this study, the data obtained from the interviews were coded, the 
data were categorized according to their similarities, the categories were associated with each other and supported 
by direct quotations from the interviews in order to understand how the pre-service teachers perceived STEM 
education, whether they adopted it or not, and whether they saw themselves as sufficient. To better understand 
the variation of the codes obtained from the qualitative interviews in three phases, they were required to use bar 
graphs to represent them graphically. In this respect, embedded quantitative analysis is included. 

For the internal validity of the research, the prepared interview form was examined by two science educa- 
tion experts, read by three pre-service teachers, and evaluated in terms of intelligibility. Necessary arrangements 
have been made. A natural environment was tried to be created during the interviews, and the answers of the 
participants were confirmed by repeating. The data obtained are presented without modification. The coding 
made by the two researchers in categorizing the data was checked, the numbers of concurrence and disagree- 
ment were defined, and the reliability of the research was checked out using the formula of Miles and Huberman 
(1994) Reliability=consensus/(consensus+disagreement). A consensus (reliability) of 93% was achieved. For the 
credibility of the interviews, the researcher spent time with the participants throughout the process. The research 
process was explained, and it was stated that the data collection was on a voluntary basis within the scope of the 
research. Instead of participants’ names codes were used. 
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Research Results 


In this study, STEM education was determined as a phenomenon, and it was tried to determine how science 
pre-service teachers perceived and adopted STEM education, whether they saw themselves as sufficient, their 
experiences, the environment and situations that affected their experiences. In the study, three-stage interviews 
were conducted with pre-service teachers. Findings from these interviews are given below. 

In the study, pre-service science teachers were first asked to define STEM in order to understand the level of 
STEM comprehension of pre-service science teachers. It was tried to understand whether there were any changes 
in the definitions of pre-service teachers before and after theoretical knowledge and after the practical knowledge. 
Seven codes were determined according to the keywords used by the pre-service teachers. The definitions, which 
were not repeated and only said by a single pre-service teacher, were discussed under the eighth code title by 
creating the “Other” code. Six pre-service teachers (PT6G4, PT8G5, PT12G3, PT25G4, PT30G4, PT32G11) defined 
STEM with more than one word after the application. Therefore, these pre-service teachers were coded twice. STEM 
definitions of pre-service science teachers are given in Figure 1. 


Figure 1 
STEM Definitions of Pre-Service Science Teachers 


1 
La 
im} 


Abbreviation Combining Integration Linking Problem Coverage of | Education Other 
of Disciplines Disciplines of Disciplines Disciplines Solving Disciplines without rote 
about Daily 
Life 
® Pre-Theoretical Knowledge -. Post-Theoretical/Pre-Application @ After Application 


At the beginning of the research, overall, the pre-service teachers were aware that the word STEM was the 
abbreviation of the initials of their discipline, but 27 pre-service teachers (41%) gave only the expansion of the ab- 
breviation without any explanation. For example, the definition of PT7G9 at the beginning of the research is“STEM 
is an abbreviation for science, technology, engineering, mathematics.” All of the pre-service teachers participated in 
STEM education practices, in which models were designed through instructions on renewable energy using edu- 
cational Lego sets before the research but did not receive theoretical information. Although STEM is not a concept 
that they first encountered, it can be said that the pre-theoretical definitions of pre-service teachers are quite 
superficial. The pre-service teachers (except for three pre-service teachers after theoretical knowledge; PT37G6, 
PT51G11, PT53G6) developed the definition of discipline abbreviation throughout the process and made changes 
in their definitions. After the application, all pre-service teachers moved away from using the term “discipline ab- 
breviation” while defining STEM. While there were two pre-service teachers who defined STEM as the integration 
of disciplines at the beginning, it increased to 17 pre-service teachers (25%) after the theoretical knowledge and 
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decreased to 12 pre-service teachers (18%) after the application. The definition of PT12G3 after the theoretical 
knowledge is as follows; “STEM is an integrated approach to science, technology, engineering and mathematics. Art and 
entrepreneurship can also be integrated” The fact that STEM integration definitions were given during the theoretical 
knowledge may have caused this situation. The decrease in the number of pre-service teachers who define STEM 
as discipline integration after the application may be due to the difficulties of pre-service teachers in disciplinary 
integration while developing an exemplary STEM activity. The majority of pre-service teachers defined STEM as 
the use of disciplines together. 15 pre-service teachers (22%) at the beginning, 27 pre-service teachers (41%) after 
the theoretical knowledge, and 22 pre-service teachers (33%) after the application explained STEM as using the 
disciplines together. For example, the opinion of PT18G4 at the end of the application; “STEM is a system created 
by the combination of science, engineering, technology, and mathematics disciplines. It is set on a project or a design 
and gives students different perspectives and creativity instincts” Although the definitions of “discipline integration” 
and “combining of disciplines” seem to be similar, pre-service teachers’ definition of “ combining of disciplines” 
was accepted that they thought that the disciplines of STEM should be given together but did not express their 
opinion on how. Discipline integration, on the other hand, has been accepted as a situation in which they have an 
idea about how the disciplines will be used. Eight (12%) at the beginning, eight (12%) after theoretical knowledge, 
and five (7%) post-practice pre-service teachers defined STEM as the “linking of disciplines”. Six (9%) after theoreti- 
cal knowledge and eight (12%) post-practice pre-service teachers defined STEM as “solving daily life problems”. 
Nine (13%) before the theoretical education, three (4%) after the theoretical education, and six (9%) post-practice 
pre-service teachers explained STEM as a top discipline that covers the disciplines. After the application, six (9%) 
pre-service teachers stated STEM as an approach that takes the students away from rote learning. PT30G4 after the 
application; “It isa program that is far from rote learning, which is formed by the coming together of branches such as 
mathematics and engineering, where students gain experience by using the topics they have learned in the theoretical 
lessons with different applications’ he defined STEM. No such definition has been found before and after theoretical 
knowledge. The definitions coded as “Other’, which were not repeated and only said by one pre-service teacher, 
were five (7%) at the beginning of the research, and two (3%) different definitions after theoretical knowledge. 
Before the theoretical knowledge, “engineering in science class’, “education with Legos’, “group work’, “student- 
centered lesson’, and “education given in private schools”; After the theoretical knowledge, definitions were made 
as “transfer of knowledge” and “open-ended course”. No different definitions were found after the application. In 
practice, no relationship was found between the group they were in and their individual definitions of the pre- 
service teachers, who formed 12 different groups. 

Pre-service teachers were asked whether they felt competent about STEM education. The responses were 
collected under three themes as being fully competent in STEM education, partially competent in STEM education, 
and not being competent in STEM education. While analyzing the data, percentage values were taken as a basis 
due to the decrease in the number of pre-service teachers who participated in the post-application interviews. An 
understanding of pre-service science teachers on STEM competencies is given in Table 2. 


Table 2 
Understanding of Pre-service Science Teachers on STEM Competencies 


Themes Codes 


Predicting what canbe Pre-Theoretical Knowledge PT13G9, PT21G10, PT24G2, PT33G1 

done in the STEM Post-Theoretical Knowledge PT13G9, PT14G6, PT21G10, PT24G2, PT29G2, PT33G1, PT34G3, PT36G1, 
PT40G2, PT43G10 
After Application PT14G6, PT24G2, PT43G10, PT55G7 


Fully Be able touse STEM _ Pre-Theoretical Knowledge PT10G6, PT44G8, PT54G5, PT60G6, 
competent Post-Theoretical Knowledge PT10G6, PT12G3, PT44G8, PT54G5, PT60G6, PT63G2 
After Application PT10G6, PT23G8, PT28G6, PT35G9, PT38G1 


Being in the middle of | Pre-Theoretical Knowledge PT64G9 
STEM Post-Theoretical Knowledge PT11G7, PT64G9 
After Application PT4G5, PT8G5, PT56G3, PT64G9 
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Feeling moderately Pre-Theoretical Knowledge PT3G10, PT9G10, PT11G7, PT14G6, PT17G9, PT20G10, PT29G2, PT36G1, 
competent PT63G2 
Post-Theoretical Knowledge PT3G10, PT9G10, PT17G9, PT20G10, PT46G9, PT48G7, PT50G4, PT66G12 
After Application PT17G9, PT20G10, PT44G8, PT46G9, PT48G7, PT50G4, PT66G12 


Finding deficiencies, Pre-Theoretical Knowledge PT2G6, PT4G5, PT12G3, PT15G10, PT18G4, PT23G8, PT25G4, PT35G9, 
feeling the need for PT39G5, PT40G2, PT41G9, PT43G10, PT49G2, PT52G4, PT55G7, PT56G3, PT57G11, PT58G2, PT59G3, 


improvement PT62G11 
Partially Post-Theoretical Knowledge PT2G6, PT4G5, PT5G7, PT15G10, PT18G4, PT19G1, PT23G8, PT25G4, 
competent PT28G6, PT30G4, PT32G3, PT35G9, PT38G1, PT39G5, PT41G9, PT49G2, PT52G4, PT55G7, PT56G3, 


PT57G11, PT58G2, PT59G3, PT61G7, PT62G11 
After Application PT2G6, PT12G3, PT13G9, PT18G4, PT19G1, PT21G10, PT22G3, PT29G2, PT32G3, 
PT34G3, PT36G1, PT39G5, PT49G2, PT52G4, PT54G5, PT58G2, PT59G3, PT61G7 


Be proficientinsome — Pre-Theoretical Knowledge PT7G9, PT26G11, PT27G8, PT30G4, PT34G3, PT50G4, PT51G11, PT53G6 
disciplines Post-Theoretical Knowledge PT6G4, PT7G9, PT16G1, PT26G11, PT27G8, PT30G4, PT45G11, PT51G11, 
PT53G6, PT65G5 

After Application PT6G4, PT16G1, PT26G11, PT27G8, PT30G4, PT33G1, PT51G11, PT53G6 


Being at the beginning Pre-Theoretical Knowledge PT5G7, PT6G4, PT8G5, PT19G1, PT26G11, PT28G6, PT30G4, PT32G3 
of STEM education PT37G6, PT38G1, PT42G7, PT45G11, PT46G9, PT48G7, PT57G11, PT61G7 
Post-Theoretical Knowledge OA8G5, PT26G11, PT22G3, PT37G6, PT42G7, PT57G11 


Not being After Application PT11G7, PT25G4, PT37G6, PT42G7, PT60G6, 
competent 


Feeling distant from Pre-Theoretical Knowledge PT1G1, PT16G1, PT22G3, PT47G9, PT65G5, PT66G12 
STEM education Post-Theoretical Knowledge PT1G1, PT47G9 
After Application PT1G1 


Three codes were formed from the data collected under the theme of being fully competent in STEM educa- 
tion. The answers of the pre-service teachers were gathered under the headings of predicting what | can do in the 
STEM field, being able to use STEM, and being in the middle of STEM. The percentage of pre-service teachers who 
stated that they were competent in STEM; 13% in pre-theoretical interviews, 27% in post-theoretical interviews, 
and 23% in post-application interviews. When the data are examined, it is understood that the percentage of pre- 
service teachers who find themselves competent in STEM increased after the theoretical knowledge but decreased 
again after the application. 

Three codes were formed from the data collected under the theme of being partially competent in STEM educa- 
tion. The answers of the pre-service teachers were grouped under the headings of feeling moderately competent, 
feeling lacking and needing improvement, and being proficient in some disciplines. The percentage of pre-service 
teachers who stated that they were partially competent in the STEM field; 56% in pre-theoretical interviews, 63% in 
post-theoretical interviews, and 63% in post-application interviews. When the data are examined, it is understood 
that the percentage of pre-service teachers who find themselves partially competent increased after the theoretical 
knowledge and did not change after the application. 

Two codes were formed from the data collected under the theme of not being competent in STEM education. 
The answers of the pre-service teachers were gathered under the headings of being at the beginning of STEM edu- 
cation and feeling distant from STEM education. The percentage of pre-service teachers who stated that they were 
not competent in the STEM field is 36% in the pre-theoretical interviews, 12% in the post-theoretical interviews, 
and 11% in the post-application interviews. When the data are examined, it is understood that the percentage 
of pre-service teachers who do not find themselves competent decreased after the theoretical knowledge. The 
percentage values before and after the application changed little. 

Examples of pre-service teachers’ views on their competencies in the STEM field are given. 

PT13G9 felt competent at the beginning of the research under the influence of the training she attended; “/ 
participated in STEM education using Lego sets. If! have similar sets, |can also practice STEM education with my students”. 
She is confident about STEM even though her statements after theoretical knowledge have changed; “By combin- 
ing STEM with entrepreneurship, | can help students see and evaluate future opportunities”. After the application, she 
stated that she had superficial knowledge; “! find myself at a superficial level and incomplete in terms of the fact that 
STEM education is new, and it isa concept and education that | have just seen”. 
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PT1G1 stated that he was far from STEM education in all interviews from the beginning of the research. His 
opinion in the post-application interview; “I find myself far from STEM education. Since it isa model, we bought from 
abroad, | think it will be difficult to adapt it. But | also accept this fact; successful scientists worked in all subjects, that 
is, in at least 4-5 fields.” 

The opinion of PT26G11, one of the pre-service teachers stated that he was competent in some disciplines; 
“| consider myself more competent in the field of science. | feel weak in the fields of mathematics and technology”. The 
opinion of OA53G6 is; “/ think !am good in science, technology and math. But | think I'm more lacking in the engineer- 
ing part in practice, we had difficulties as a team in the engineering discipline.” 

In order to understand whether pre-service science teachers have adopted STEM education, they were asked 
if they believed in STEM education. All of the pre-service teachers stated that they believed in STEM education in all 
three stages and answered the question by explaining the possible contributions of STEM education. The answers 
given by the pre-service teachers and their answers to the possible contributions of STEM education were examined 
under two themes as contribution to the student and contribution to the society. The codes formed under the 
theme of contribution to the student were divided into categories under the title of cognitive change, affective 
change and change in skills. No categories have been created for codes under the theme of contribution to soci- 
ety. The themes, codes and categories of the data obtained in all three stages of the research are given in Table 3. 


Table 3 
Themes, Codes and Categories Obtained from The Understanding of Pre-Service Teachers About the Contributions of STEM 
Education 


Theme Category Codes 


Skill Development 


21st Century Skills 


Problem Solving Skill 


Creativity 


Innovative Idea / Innovation 
Change In Skills 


Entrepreneurship 


Critical Thinking 


Collaboration 


Communication 


Contribution 


to the student Design Skill 


Increases Achievement, Knowledge Increase and Transfer 


Persistence in Knowledge, Avoids Rote Learning 


Diverse/Interdisciplinary Perspective 


Cognitive Change 
ee 9 Increase in Science Knowledge 


Increase in Technology Knowledge 


Increase in Engineering Knowledge 


Increase in Mathematics Knowledge 


Increased Curiosity 


Affective Change 
Develops Imagination 


Individuals Compatible with The Developing World 


Contribution It Meets the Need of Qualified/Producer People 
to Society 


Connects Daily Life Problems 


Affects Career Choice 
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The first category created under the theme of contribution to students is the category of change in skills. 
Pre-service teachers stated that STEM activities would change students’ skills in all three stages. In cases where it 
was stated that STEM activities would contribute to students’ skills but did not specify what these skills were, the 
data were collected under the “skill development” code. Similarly, the data stating that STEM activities contribute 
to students’ 21st-century skills and that these skills are not specified are also stated as “21st-century skills” were 
collected under the code. 51 pre-service teachers before the theoretical knowledge, 14 pre-service teachers after 
the theoretical knowledge, and one pre-service teacher after the application stated that STEM activities would 
improve the skills. For example, before the theoretical knowledge, PT33G1 explained the contribution of STEM 
as “STEM education allows students’ skills to develop.”. After the theoretical knowledge, “STEM education brings the 
characteristics of individuals such as critical thinking, problem-solving, being creative and self-directed.” stated the skills 
by name. After the application, justified it as “/ believe in STEM education because it will enable students to solve the 
problems they encounter in daily life, design projects, develop their cognitive development and develop their imagina- 
tions.” There was no pre-service teacher who used the statement that “STEM education contributes to 21st-century 
skills" before theoretical knowledge. Seven pre-service teachers used the term 21st-century skills after the theoreti- 
cal knowledge and five after the application. The fact that the number of participant pre-service teachers was 66 
and none of them used the expression at the beginning of the research made us think that pre-service teachers 
did not associate 21st-century skills with STEM until the theoretical knowledge. 

According to pre-service teachers before theoretical knowledge, the skills that STEM education leads to change; 
Collaboration (87%), designing (75%) and innovation (31%) skills. After the theoretical knowledge, according to 
the pre-service teachers, the skills that STEM education leads to change; Problem-solving (45%), creativity (18%), 
innovation (16%), entrepreneurship (13%), critical thinking (6%). After the application, according to the pre-service 
teachers, the skills that STEM education leads to change; Problem-solving (44%), innovation (32%), creativity (23%), 
cooperation (5%), entrepreneurship (7%), critical thinking (7%), communication (3%) skills. 

In the pre-theoretical interviews, problem-solving skills were never mentioned by the pre-service teachers, but 
the ability to design was indicated with a percentage of 75%. It is thought that this difference.is because of the fact 
that the participant pre-service teachers participated in STEM education applications, in which models related to 
renewable energy were designed using educational Lego sets, and that they completed the projects in line with the 
instruction. PT20G10’'s opinion before the theoretical knowledge; “Of course | embrace STEM education. STEM is more 
educational and useful. For example, Students will be able to design renewable energy themselves with Lego sets and learn 
by using their imaginations." supports this idea. Problem-solving skills are at the forefront of the skills that pre-service 
teachers think STEM education will contribute to students after theoretical knowledge and practice. This may be 
due to the introduction of the Problem-based STEM model during theoretical knowledge, and the majority of the 
groups planning STEM activities for this strategy in practice. After the theoretical knowledge, PT50G4 explained the 
contributions of STEM to the student through the problem-based STEM model; “With the problem-based STEM ap- 
proach, students can find solutions to the problems they encounter in their daily life and do research.. The post-application 
opinion of PT11G7 was “The biggest shortcoming of the students is that they do not use the information they have learned 
in daily life. | intend to develop students with a problem-based STEM approach.’ is in the form. 

It is quite surprising that although cooperation skill is mentioned first in the list of skills changed by STEM 
education before the theoretical knowledge, it is not emphasized after the theoretical knowledge and is mentioned 
by very few pre-service teachers after the application. The pre-service teachers completed their activities by using 
instructions in the STEM study they participated in before the research. Knowing the product that will be formed 
in the directive projects at the beginning may have made the cooperation easier. In this study, pre-service teachers 
worked with the group to create their own projects like students. Pre-service teachers may have had difficulty col- 
laborating in completing the open-ended STEM activity that required them to come together on acommon point. 

Both theoretical knowledge and post-practice pre-service teachers stated that STEM education will cause a 
change in students’ creativity and innovation skills. Similarly, it is believed by pre-service teachers that entrepre- 
neurship and critical thinking skills will contribute. After theoretical knowledge PT59G3; “! believe in STEM educa- 
tion because STEM brings critical thinking, creativity, problem-solving and entrepreneurship in individuals.” she stated. 

The second category created under the theme of contribution to students is the cognitive change category. 
Pre-service teachers expressed their opinions on cognitive change with high percentages before the theoretical 
knowledge. They stated that there would be an increase in students’ knowledge of each discipline that makes up 
STEM. At the beginning of the research, pre-service teachers stated that STEM education would increase students’ 
technology knowledge (80%), engineering knowledge (78%), science (78%) and mathematics knowledge (78%). 
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However, this high percentage of consensus has not been seen after theoretical knowledge and application. The 
percentage of pre-service teachers who state that STEM will increase their technology knowledge is as low as 6% 
after theoretical knowledge and 5% after practice. While the opinion that there will be an increase in engineer- 
ing knowledge was not encountered after the theoretical knowledge, it was expressed with a rate of 9% after 
the application. There was no pre-service teacher who expressed the opinion that there would be an increase in 
knowledge in science and mathematics disciplines after the theoretical knowledge and application. These find- 
ings showed that 14 weeks of interactive STEM education made a significant change in the views of pre-service 
teachers. Pre-service teachers initially interpreted STEM more superficially. “Since it is mentioned in the name of 
science, technology, engineering and mathematics disciplines, there will be an increase in knowledge in these fields.” 
interpreted as. However, the theoretical and practical training provided has led to a change in this way of think- 
ing. Pre-service teachers stated that STEM education provided students with an interdisciplinary perspective; after 
theoretical knowledge (33%) and after practice (28%). Relatedly, the percentage of opinions that STEM education 
will increase students’ success and increase their knowledge is before the theoretical knowledge (48%), after the 
theoretical knowledge (24%) and after the application (7%). Opinions on the permanence of the knowledge; It is 
a decreasing percentage at the beginning of the study (78%), in the middle (10%), and at the end (5%). While pre- 
service teachers thought of STEM education as a miraculous approach to cognitive development at the beginning 
of the research, they did not think in the same way after the application phase, which helped them think like a 
student and formed more cautious statements. For example, PT48G7's opinion on STEM at the beginning of the 
research; “/ would ensure that STEM education is implemented in every school. In this way, the next generation would 
be more knowledgeable and cultured, while the knowledge given by heart is forgotten over time, the knowledge learned 
with STEM becomes permanent.’ is in the form. If we look at the view of PT48G7 after the theoretical knowledge, a 
more cautious point of view is realized; “STEM education is an application that will be useful in school. It will enable 
students to transfer knowledge. Time is needed for our country, as it is just beginning to settle down.” The opinion of 
PT48G7 after the practical training is; “STEM can contribute to the development of many aspects of students such as 
conceptual learning skills and 21st-century skills. It can help students to look at the subject from all angles, not just take 
the knowledge, but question and research it.” 

Four codes were determined under the theme of contribution to society. These codes are 1. raising individuals 
who are compatible with the developing world, 2. meeting the needs of qualified and productive people, 3. raising 
individuals who are aware of daily life problems, and 4. affecting the career choice of students. There was no pre- 
service teacher who expressed an opinion on each of the four code headings before the research. The percentage 
of pre-service teachers who gave their opinions on the topics after the theoretical knowledge decreased after 
the application. It is believed that this is due to the fact that developed country STEM goals are given during the 
theoretical knowledge. The decrease after the implementation may be that they think that it will not be so easy to 
reach the said goals. Percentages of pre-service teachers’ opinions after theoretical knowledge; raising individuals 
who are compatible with the developing world (27%), meeting the needs of qualified and productive people (36%), 
raising individuals who are aware of daily life problems (21%), and affecting the career choice of students (12%). 
Percentages of pre-service teachers’ opinions after the application; raising individuals who are compatible with the 
developing world (no opinion), meeting the need for qualified and productive people (13%), raising individuals 
who are aware of daily life problems (19%), and affecting the career choice of students (1%). A few examples from 
the views of pre-service teachers are as follows. After the theoretical knowledge PT5G7; “STEM will benefit not only 
education but also many other fields, both in Turkey and in the world, and will be a pioneer in the development of societ- 
ies. With the combination of more than one discipline, a much better product will emerge and there will be well-trained 
and qualified individuals in every field.” she said. The opinion of PT8G5 is; “He knows how to use his knowledge skills in 
the tasks that a person has to do in his daily life and in the problems that need to be solved. In the simplest way, a child 
can make a fountain to be built in the garden of their house together with his father” 

In the interviews at the end of the research, the pre-service teachers were asked “whether there were any 
difficulties in the implementation phase” and, if so, “what happened”. In addition, how should STEM education 
be given to teacher candidates? The question has been posed. Pre-service teachers stated that STEM discipline 
integration is difficult (34%) and they had difficulties in performing STEM activities due to insufficient engineer- 
ing education (50%). During the application, pre-service teachers wanted to use sensor sets in their projects and 
they wanted to use “Arduino”, one of the programming platforms suitable for this and is thought economical and 
the programming language is easy. However, they stated that they had difficulties in this adventure and because 
they did not know coding (50%), they received support from their friends studying in the software engineering 
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department. They stated that STEM education should be given as a course during their undergraduate education 
(67%). Before the training, it was stated that in order for the pre-service teachers to adopt STEM education, their 
contributions should be explained (11%) and that they should be associated with daily life (7%). 

For PT24G2, the part where she had a hard time was; “We used Arduino in our project, and it was not something 
| had seen before in my life. It was difficult to work with unfamiliar materials and it was a difficult process in the coding 
part”. Regarding providing STEM education to pre-service teachers, “They must have a readiness before coming to 
university. STEM should be applied in other education levels, albeit to a lesser extent. Starting from the first grade, the 
pre-service teacher should see engineering in addition to the field courses, know technology well, and STEM applied 
courses should be given’ For PT27G8; “We had a hard time integrating engineering and technology. We got help from 
our friends studying in the software engineering department” The opinion of PT61G7 is; “During this process, | had the 
chance to do inquiry. | saw that teachers defended the STEM understanding by doing simple repetitive activities that 
they saw on the internet. They open various courses about STEM that have no theoretical foundation, but | think they 
are of no use to students. The main thing is to make the student think about STEM. Therefore, the contributions of STEM 
should be explained to pre-service teachers, and they should be adopted.” 

In all three stages, the pre-service science teachers were asked which STEM learning and teaching models 
they preferred to use. At the beginning of the research, it was understood that pre-service teachers did not adopt 
a certain STEM learning and teaching model. They answered the question by giving examples from the activities 
they did using the Lego sets they participated in before the research. | use Lego sets if available at my school (25%), 
| build a wind/solar car (42%); | can’t do without STEM materials (18%), | have no STEM understanding (15%). It 
was thought that the pre-service teachers could not make sense of the question because STEM theory knowledge 
was not given in the training they attended before the research. After the theoretical training, the answers of the 
pre-service teachers were distributed as follows: Four different models were specified: Problem-based STEM (43%), 
5E integrated STEM (45%), project-based STEM (13%), and design-based STEM (9%). Seven pre-service teachers 
(PT8G5, PT17G9, PT27G8, PT31G5, PT39G5, PT41G9, PT52G4) stated more than one option. During the theoretical 
knowledge, learning and teaching models used in the literature were introduced, and access to the sources with 
sample applications was provided. The fact that pre-service teachers stated a STEM learning model compared to 
the beginning of the study showed that it is important to give STEM theoretical knowledge. In the interviews made 
after the STEM applications, the most preferred learning model was Problem-based STEM (63%). Project-based 
STEM (23%) ranked second. Eight pre-service teachers stated that a single STEM learning and teaching method is 
not correct and emphasized that different models should be used depending on the situation (OA9G10, PT21G10, 
PT24G2, PT28G6, PT43G10, PT46G9, PT56G3, PT58G2). One pre-service teacher stated the design-based STEM 
(PT29G2), 5E integrated STEM (PT52G4), and Science-based STEM (PT4G5) models as the adopted model. During 
the implementation process, pre-service teachers performed 11 group and one individual STEM activities (Table 
1). Seven of these activities were prepared using the problem-based STEM, four of them using the project-based 
STEM, and one using the design-based STEM model. According to the data obtained, it can be said that the teacher 
candidates adopt the problem-based STEM model more. Examples of pre-service teachers’ opinions; 

The opinion of PT66G12, who worked individually during the implementation phase; “/ had prepared my 
application activity according to the project-based model. But problem-based thinking is actually better. It would be a 
priority for me to put more creative thinking towards life’s problems and put it into practice in life.” 

PT34G3; “We need to analyze a certain problem situation and find various solutions in line with these problems, 
so! think problem-based STEM would be a better model.” 

PT46G9’s opinion; “Our group work was oriented towards a problem-based STEM model. However, any STEM 
model that the person considers self-sufficient can be used. Model selection may vary depending on which teaching 
level it is used and for a better realization of the solution and subject. | think the problem-based STEM model is more 
advantageous because it leads to problems that exist in daily life.” 

PT23G8; “! adopt the project-based STEM model because new products must be obtained to make life easier.” 


Discussion 
In the study, pre-service science teachers were asked to define STEM at three different times in order to de- 
termine pre-service teachers’ understanding of STEM and whether their understanding changed according to the 


training given in the process. The definitions of pre-service science teachers are gathered under seven codes under 
the headings of abbreviation of disciplines, integration of disciplines, combining of disciplines, coverage of disci- 
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plines, problem-solving about daily life, linking of disciplines, and education without rote. In addition, there were 
five different definitions at the beginning and two different definitions in the middle of the study by pre-service 
teachers. Eight conceptualized integrated STEM education models were proposed in the study conducted by Ring 
et al. (2017). Three of these models, an abbreviation of disciplines, integration of disciplines, and problem-solving 
from daily life, are similar to the definitions of pre-service teachers in this study. Dare et al. (2021) found in their re- 
search that teachers have a STEM education perspective in which real-world problems are used after participating 
in STEM-focused professional development and performing integrated STEM lessons in their classrooms. Sarioglu et 
al. (2022), the definitions of interdisciplinary approach and understanding real life problems came to the fore in the 
study they conducted with teachers who had participated in various STEM trainings before. It showed parallelism 
with the definition of “problem-solving from daily life” obtained in this study. In this research, STEM was expressed 
by pre-service teachers as “solving real-world problems” after theoretical knowledge and application. STEM defini- 
tion accepted by more pre-service teachers at the end of the research; definitions of “combing of disciplines“ and“ 
integration of disciplines” At the end of the research, pre-service teachers moved away from the non-explanatory 
definition of “the abbreviation of science, technology, engineering and mathematics disciplines” Pre-service science 
teachers changed their definitions of STEM during the theoretical and applied education process they attended. 
However, there is no single definition that all pre-service teachers agree on. The differences in the definitions of the 
pre-service teachers in the same group showed that STEM was not interpreted the same by the pre-service teachers. 
According to the studies (Bybee, 2013; English, 2016; Johnson, 2012; Paz et al., 2022) showing that there is no single 
definition of STEM, this situation was not surprising. However, the point to be discussed is that despite participating 
in the same theoretical and practical training, STEM education, which is perceived differently, is that the students of 
pre-service teachers who will do STEM applications with their own students in the future will encounter completely 
different applications under the name of STEM. Although conceptualizing a common STEM model is difficult, a 
clearer environment can be created (Breiner et al., 2012). 

Pre-service teachers were asked whether they felt competent in STEM education, and the answers were grouped 
under three themes: being fully competent in STEM education, partially competent in STEM education, and not be- 
ing competent in STEM education. It was understood that the majority of pre-service teachers think of themselves 
as partially competent. It is understood that the percentage of teacher candidates who find themselves competent 
increased after the theoretical knowledge but decreased again after the application. In parallel with this research 
result, Bartels and Rupe (2019) found that pre-service teachers’ understanding of STEM did not increase even after 
they planned and implemented STEM lessons. This may be because they realize that the points that seem feasible 
with theoretical education are not as expected in applied education. Teachers’ participation in STEM teaching can 
be supported by improving their self-efficacy in STEM practices (Dong et al.,2019; Shahali et al., 2015). It is thought 
that pre-service teachers will be more conscious while guiding their students in the future with the proliferation of 
STEM applications in which they participate as students. Many studies with teachers and pre-service teachers stated 
that they needed courses and in-service training to improve themselves in the field of STEM (Aydeniz, 2017; Yildirim 
et al., 2022; Yildiz, 2023). The decrease in the number of pre-service teachers who do not find themselves competent 
after the theoretical knowledge has shown that the training to be given, even at the theoretical level, will contribute 
to the STEM competence of the pre-service teachers. This result is in line with the studies that the courses or train- 
ings provided contribute to the STEM competencies of teachers (Arslanhan & inaltekin, 2020; Dong et al., 2019). 

In the research study, the understanding of the pre-service teacher, who stated that he was far from STEM 
education because it was a foreign-sourced model, was found to be very important. Cepni (2017) stated that some 
mistakes were made in the process of spreading STEM education in Turkey, such as giving STEM certificates with a 
few-day courses, calling hobby courses STEM, and assuming that STEM can only be done with expensive and robotic 
sets. Although there are some studies carried out in education faculties programs, the absence of STEM courses at 
the undergraduate level requires pre-service teachers to improve themselves with their personal efforts at this point 
and to find their own truths in information pollution. 

Pre-service teachers stated that they believed in STEM education and explained their possible contributions. 
The fact that pre-service teachers do not actively teach and have never tested STEM with students is the inference 
of the possible contributions they have stated as a result of their experiences in their own education processes. 
The answers of the pre-service teachers regarding the possible contributions of STEM education were examined 
under two themes as contribution to the student and contribution to the society. According to pre-service teach- 
ers, STEM education causes positive changes in students’ skills and cognitive and affective development. There are 
many studies about STEM education, that confirm the inferences of pre-service teachers, contribute to the cogni- 
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tive development of students (Fan & Yu, 2017; Wendell & Rogers 2013), contribute to their affective development 
(Bakirci et al., 2022; Kegeci et al., 2017), contribute to students’ skills (Sahin & Top, 2015; Zengin et al., 2022), studies 
that affect career choices (Chachashvili-Bolotin et al., 2016; Tanenbaum, 2016). In this research, problem-solving 
and creativity skills were stated as the leading skills that STEM education contributed by the pre-service teachers. 
Interestingly, collaboration and design skills were stated by the majority of pre-service teachers before the research, 
but the pre-service teachers who advocated different models had problems in cooperation while working on a 
project by meeting on a common point. The collaboration skill was emphasized by very few pre-service teachers at 
the end of the process. Pre-service teachers who advocated different models had problems in cooperation in this 
research, where they met on a common point and completed a project. Collaboration may have been easy for pre- 
service teachers in STEM applications, which they designed based on the instructions they participated in before 
the research. In this case, it can be concluded that the method followed in STEM applications and the skills affected 
at the end of the process will vary. 

Pre-service teachers’ understanding about the contribution of STEM education to society is raising individuals 
who are compatible with the developing world, meeting the needs of qualified and productive people, raising in- 
dividuals who are aware of daily life problems and making connections, and affecting the career choice of students. 
These results are consistent with the targeted outcomes in STEM reports (MNE, 2016; NRC, 2012; TUSIAD, 2014). 
There were no pre-service teachers who expressed similar views before the research, it increased after the theoreti- 
cal knowledge and decreased after the application. It is believed that this is due to the fact that developed country 
STEM goals are given during the theoretical knowledge. The decrease after the implementation may be that they 
think that it will not be so easy to reach the said goals. 

In the application part of the research, pre-service teachers prepared a STEM activity that they could apply with 
their students in the future, and a lesson plan on how to implement the activity, according to a STEM model they 
chose with their group friends. The pre-service teachers stated their science, technology, engineering and math- 
ematics achievements in their activity plans and presented to their friends how they integrated the disciplines and 
which STEM approach they adopted. Pre-service teachers had a very difficult time in STEM discipline integration. 
Vasquez et al. (2013) stated that integration can be used at various levels such as transdisciplinary, interdisciplinary, 
multidisciplinary and disciplinary. The difficulty of pre-service teachers may be that they cannot choose from this 
diversity. Dare et al. (2019) stated that teachers have a limited understanding of what STEM is and what it means for 
their teaching, despite all the guidance. Alan et al. (2019), in their study that aimed to support pre-service science 
teachers’integrated teaching knowledge through STEM applications, found that pre-service teachers believed in the 
necessity of STEM education, but thought that discipline integration was not easy. Pre-service teachers stated that 
they had the most difficulties in the implementation process due to the inadequacy of their engineering education. 
Except for two groups, other groups wanted to use sensor sets in their projects and wanted to use “Arduino”, one 
of the programming platforms suitable for this. However, they stated that they received support from their friends 
studying in the software engineering department because they did not know coding. It has been stated in many 
studies that pre-service teachers think that they are inadequate in engineering and that they do not trust themselves 
(Akgtinduz et al., 2015; Avsec & Sajdera, 2019; Aydeniz & Bilican, 2017; Aydeniz & Cakmakci, 2017; Blackley & Howell, 
2015). Pre-service science teachers stated that engineering and STEM education should be given as a course in 
undergraduate education, especially coding learning is necessary. Aydeniz (2017) stated that students studying in 
science and mathematics departments should take at least two practice-oriented coding courses. When it comes 
to engineering content integration, it is thought that software engineering comes to the fore because most of the 
current technologies require coding and the spread of STEM education in Turkey is mostly through robotic sets. 

Pre-service science teachers adopted the problem-based STEM model among STEM learning and teaching 
models and preferred to use it. Pre-service teachers emphasized problem-solving in their STEM definitions and 
contributions questions. It was an expected result that the STEM learning model they chose was a problem-based 
model. 5E integrated STEM, project-based STEM, design-based STEM, and science-based STEM models were also 
shared by pre-service teachers. Yildirim (2018) stated STEM models as project-based learning, inquiry-based learn- 
ing and problem-based learning models in his research. In this research, project-based STEM was preferred in the 
second place. Eight pre-service teachers stated that a single STEM learning and teaching method is not correct and 
emphasized that different models should be used depending on the situation. There is no single model in STEM 
education with which consensus has been established (Dare et al., 2019; Sarioglu et al., 2022; Selvi & Yildirim, 2017). 
Therefore, the important thing is that teacher candidates know the models, have a good command of STEM educa- 
tion and can apply any model they want. 
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Conclusions and Implications 


In the study, in which STEM education was determined as a phenomenon and how the pre-service science 
teachers adopted it, what the understanding of pre-service teachers’ STEM education was, whether they thought 
of themselves as sufficient, and their predisposition to STEM teaching and learning models, valuable results were 
obtained. The results help us to look at STEM education from the perspective of pre-service science teachers. 
There is no single definition that pre-service science teachers agree on. Despite their collaborative work during 
the implementation phase, the pre-service teachers did not unite in a common definition, and each pre-service 
teacher expressed their own definition of STEM. Most of the pre-service science teachers defined STEM as “the ab- 
breviation of disciplines” without going into details at the beginning of the research, but they moved away from this 
definition after theoretical knowledge and practical training. STEM definitions prominent by pre-service teachers 
are; “combining of disciplines’, “integration of disciplines” and “solving daily life problems”. 

Today, when we are about to complete the first quarter of the 21st century, it is very important to increase 
the competencies of our teachers who will raise the generations that will deal with much bigger problems in the 
future. The number of pre-service science teachers who did not consider themselves competent at the beginning 
of the study decreased after the theoretical and practical training. The number of science teachers who consider 
themselves competent has increased. However, some of the pre-service teachers who felt fully competent after 
the application defined themselves as moderately competent. Although the majority of pre-service teachers 
participated in theoretical and practical training, they defined themselves as intermediate level competency in 
STEM education. The trainings provided contributed to the STEM competencies of pre-service science teachers. As 
stated by the pre-service teachers, the self-confidence of the pre-service teachers can be increased by providing 
the trainings throughout their undergraduate education. 

Participating pre-service science teachers believe in STEM education. They explained the reason for their 
belief through the possible contributions they would make to the students. According to pre-service teachers, 
STEM education contributes to both students and society. Pre-service science teachers had the most difficulty in 
disciplinary integration during the STEM theory and practice education process they attended. The pre-service 
teachers stated that their engineering knowledge was lacking and stated that they should especially receive coding 
training. The majority of pre-service science teachers preferred to use the problem-based learning model among 
STEM learning and teaching models. Some pre-service teachers stated that a single model would not be correct 
and that different models should be used depending on the situation. 

Unlike the studies examining the understanding of pre-service teachers about STEM education, in this study, 
it was tried to determine the changes in pre-service teachers’ understanding of STEM during the STEM education 
process. 
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Introduction 


The OECD (2022) defines Al systems as machine-based entities capable 
of making predictions, recommendations, or decisions influencing real or 
virtual environments, aligned with human-defined objectives. Many educa- 
tors and professionals remain unaware that national Al education policies 
were established even before ChatGPT’s emergence, a milestone in Al's 
educational integration. The OECD began addressing this as early as 2021 
(Galingo et al., 2021), summarizing various national strategies. Notably, Al 
educational guidelines, such as “Is education losing the race with technol- 
ogy” (OECD, 2023) and“OECD digital education outlook” (OECD, 2021), were 
established before ChatGPT’s advent. The European Commission has also 
contributed with publications like “Ethical Guidelines on the use of artificial 
intelligence (Al) and data in teaching and learning for Educators” (European 
Commission, 2022) and “White paper on artificial intelligence — a European 
approach to excellence and trust” (European Commission, 2020). However, 
until Open Al popularized ChatGPT in November 2022, these documents 
seemed futuristic. Now, they hold immediate relevance. Natural language 
processing tools, exemplified by ChatGPT, have revolutionized education. 
This Al-driven model assists in diverse tasks, from coding to essay writing. 
As a result, the educational sector is undergoing significant transformation. 
The predictions of Seldon and Abidoye (2018) in “The Fourth Education 
Revolution” have rapidly materialized. The sudden emergence of such tools 
found many educators unprepared, leading to polarized views on Al's role 
in education. Regardless of perspective, Al’s ubiquity necessitates strategic 
adaptation. Some nations have even implemented regulations or outright 
bans on tools like ChatGPT, emphasizing the need for ethical engagement 
with Al in education (Bhati, 2023; McCallum, 2023; Yang, 2023). Cotton et al. 
(2023) highlight the dual nature of Al in education, presenting both chal- 
lenges and opportunities. Educational assessment serves multifaceted pur- 
poses, with its nature diverging based on the chosen method. Criteria such 
as construct validity, reliability, desired impact, and resource optimization 
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are essential for evaluating assessment techniques. Teacher assessment remains central to these criteria (Harlen, 
2007). Conventional assessment methods, primarily paper-based, are familiar to educators (Montgomery, 2002), 
however, these methods are characterized by rigid question formats, offer a limited snapshot of an individual’s 
capabilities often sidelining multifaced learning perspectives (Frey & Schmitt, 2010), and do not cater to stu- 
dents’ unique knowledge, skills, and experiences and evaluate competencies typically beyond computerization 
(Swiecki et al., 2022). While teachers have incorporated technology to support conventional assessment practices 
(Grion et al., 2018) and have adapted their pedagogical and evaluative strategies during exigencies (Legvart et 
al., 2022; Turan-Guntepe et al., 2023), Al’s advent in grading introduces novel dimension. Al’s inherent nuances 
necessitate educators’ cognizance during evaluation, transcending mere performance measurement. Educators 
must contemplate assessment’s overarching role and objective, as it significantly influences both teachers’ and 
students’ value perceptions. Such reflection enables judicious student evaluation, catering to the varied expecta- 
tions of stakeholders through the assessment trajectory and its outcomes (Dunn et al., 2004, p. 15). Yet, educators 
must remain vigilant of A’s constraints, exemplified by ChatGPT, which is characterized by post-September 2021 
knowledge gaps, misinformation, comprehension lapses, and hallucinations, potentially generating spurious 
data. Such scenarios, exacerbated by a failure to align with children’s development nuances, risk jeopardizing 
their well-being. This challenge becomes pronounced with Al not explicitly designed for educational purposes, 
especially concerning younger students. This predicament starkly contrasts with the child-centric Al principles 
delineated by UNESCO (Miao et al., 2020, p. 4), which has underscored child protection from Al's adverse impacts, 
child agency in Al system shaping, and the judicious harnessing of Al's potential benefits. 

While teachers may not have direct experience with assessment in the emerging landscape of artificial in- 
telligence in education, parallels can be drawn with their experiences during remote teaching. The transition to 
remote teaching, necessitated by global circumstances, required educators to swiftly adapt and integrate technol- 
ogy into their assessment practices. This rapid shift, much like the introduction of Al in education, presented both 
challenges and opportunities for educators. Experiences from the period of emergency remote teaching have 
indicated that while traditional assessment methods possess inherent advantages, they may not fully capture the 
breadth and depth of learning and teaching in technologically advanced environments. The integration of Alinto 
assessment necessitates a re-evaluation of existing practices. The parallel, link, and interplay between distance 
learning and artificial intelligence are explored in numerous studies (e.g., Aljarrah et al., 2021; Tang et al., 2023). 
These developments call for modifications in teaching methodologies, assessment practices, and the incorpora- 
tion of digital tools, emphasizing the critical role of digital literacy competencies. The shift to remote teaching 
provides insight into teachers’ adaptability and potential readiness for the integration of Al-driven assessment 
methods. Recent research further supports the perspective. Kim and Kim (2022) highlighted the significance 
of teachers’ perceptions of Al-enhanced tools in STEM education, suggesting that successful Al integration is 
influenced by teachers’ attitudes and prior experiences. Kerneza and Zemljak (2023) posited that teachers have 
preconceptions about future technologies, such as humanoid robots and Al, that influence their perceptions of 
these technologies. Salas-Pilco, Xiao, and Hu (2022) examined the relationship between Al and learning analytics 
in teacher education, emphasizing the importance of understanding teachers’ digital competence and their views 
on Al's role in teaching. Assessment in remote settings frequently involves the utilization of digital tools, many of 
which are intrinsically linked to artificial intelligence. Understanding how teachers employ and adapt to remote 
assessment techniques can serve as an indicator of their readiness and capability to integrate Al into evaluation 
processes. Furthermore, their receptiveness to adopting new technologies within the context of remote assess- 
ment may reflect a broader preparedness for innovations, such as the incorporation of artificial intelligence in 
educational practices. Given these insights, it is justified to generalize teachers’ feedback on distance education 
to the broader context of Al-driven assessment. The shared challenges and opportunities in both areas, informed 
by recent research findings, offer a comprehensive understanding of teachers’ preparedness for the evolving 
landscape of Al in education. 


Research Problem 
The educational landscape is dynamically evolving in real time. The OECD (2018) highlighted the unparalleled 
advancements in science and technology, especially in biotechnology and Al, as pressing challenges for the future 


of education. Among the multifaced dimensions of Al in education, the rise of ChatGPT, alongside other current 
and forthcoming generative pre-trained transformers and large-scale language models, draws particular attention 
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to the assessment and unique features that advanced technologies bring. Following initial restrictions on ChatGPT 
in specific countries, there is a growing consensus that chatbots should not be banned, but rather that students 
should be taught how to use them responsibly (Crawford et al., 2023; Gimpel et al., 2023). The discourse under- 
scores the pillars of integrity, ethics, and personal responsibility, emphasizing that individuals are responsible for 
the quality of their work (Rudolph et al., 2023). There's also a call for discerning utilization and understanding the 
inherent boundaries of non-human text generators, with a spotlight on the intrinsic worth of human composition 
(Anson & Straume, 2022; D'Agostino, 2022; Fyfe, 2022), which remains a cornerstone of intellectual growth (Mills, 
2023). Currently, there’s a push for assessment methodologies that prioritize oral presentations, self-reflection, 
performance-based assessment, and peer assessment, all underpinned by collaborative work (Gimpel et al., 2023; 
Rudolph et al., 2023). The incorporation of mentorship and coaching, which segment learning into smaller pieces 
and provide more feedback, is perceived as beneficial (Gimpel et al., 2023). Indeed, this pivotal moment may mark 
a paradigm shift from conventional pedagogical approaches towards not just innovative, but deeply transforma- 
tive instructional strategies. 


Research Focus 


In the realm of pedagogical advancements, the deployment of intricate, non-traditional assessment tools 
becomes imperative. Specifically, within the curriculum framework, science teachers ought to plan assignments 
that encourage students to think critically (RUUtmann, 2019). Furthermore, teachers should review assignments and 
assessments in their courses (Teaching in the Al era, 2023). The initial stride toward this objective was catalyzed by 
emergency remote teaching (Khan et al., 2021). The next phase should include the integration of new Al technolo- 
gies. The subsequent phase should encompass the seamless integration of emergent Al technologies. Contrary to 
the procedural aspects of Al-driven assessment, such as automated grading as described by Gardner et al., (2021), 
the primary research emphasis pivots towards the evaluation of knowledge, both facilitated by and rooted in Al. 
The deployment of Alin this context can manifest in concealed, explicit, or intentional forms, contingent upon the 
educator's strategic choices. The present study draws from teacher feedback collated at the end of the emergency 
remote teaching phase, wherein they elucidated the assessment modalities employed during remote instruction. 
Distance remote teaching has indeed opened the door to unconventional forms of learning and assessment, such 
as scalability, research innovation, flexible learning, diversity, adaptation of assessment methods, and potential 
for innovation (Gurajena et al., 2021). The primary research focus was to examine teachers’ readiness to embrace 
novel assessment methods resulting from the application of Al. It seeks to understand the challenges teachers 
may encounter when evaluating novel environments, including those related to reliability, fairness, and objectivity 
of assessment using Al. 


Research Aim and Research Questions 


The research aim was to explore the preparedness of science teachers for assessment in the age of Al, consid- 
ering the challenges and opportunities presented by the widespread accessibility of Al technologies in education. 
One of the major goals of this study was to understand the specific factors contributing to teachers’ readiness or 
their lack thereof. Furthermore, the study had an objective to discern potential differences or similarities in the 
preparedness of science teachers compared to teachers in other subject areas. Through these detailed objectives, 
the overarching aim was to offer insights into teachers’ readiness for Al-based assessments, thereby guiding edu- 
cational practices and policies in adapting to the evolving landscape of technology in education. The following 
research questions were formulated: 

RQ1: Are teachers ready for assessment in the era of widespread accessibility of Al in education? 

RQ2: Are there differences between science teachers and teachers of other subjects in terms of assess- 
ment in the era of widespread accessibility of Al in education? 

RQ3: Is assessment in the era of broad accessibility of Al in education based on novel models that require 
higher levels of reading literacy? 
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General Background 


In the contemporary educational landscape, the integration of artificial intelligence (Al) is reshaping learn- 
ing and teaching paradigms. The broader context of this transformation underscores the importance of adapting 
assessment strategies to these technologically advanced environments. A review of existing literature reveals a 
consensus on the need for novel approaches to learning, teaching, and assessment that align with 21st-century 
skills. However, a notable gap emerges when juxtaposing this consensus with the current state of teacher pre- 
paredness. Many educators, despite the wealth of research advocating for change, remain inadequately equipped 
to navigate the intricacies of assessment in an Al-augmented setting. This disparity between the recognized 
educational imperatives and the actual capabilities of teachers is further accentuated with the rapid proliferation 
of Al in educational settings. The urgency of this matter becomes even more pronounced when considering the 
responses from educators during the emergency remote teaching phase. During this period, the transformative 
potential of Al, rather than being an integral part of the educational process, was often perceived as a distant, fu- 
turistic concept. The study draws on data collected during the period of emergency remote teaching, presenting a 
unique context for understanding how teachers adapt and respond to technological shifts. This specific timeframe 
allows us to comprehend assessment practices under exceptional circumstances, which can serve as a foundation 
for understanding teacher readiness and needs in an era of rapid technological advancement, such as the integra- 
tion of Al. The intention is to offer insights that can shape practices, policies, and further research, ensuring that 
the educational community can harness the full benefits of Al while addressing its challenges. 


Sample 


The survey is based on a questionnaire completed by 1215 teachers from primary and secondary schools in 
Slovenia. The sample size of 1215 was determined based on a power analysis to ensure adequate statistical power 
for detecting meaningful differences and associations in the data. A power analysis was conducted to determine 
the appropriate sample size. The power analysis was performed considering an anticipated effect size of d = .5 
(Cohen, 1988). The standard deviation of responses, obtained from a pilot study, was also considered and found 
to be 1.2. The calculation was carried out using the SPSS software, an aim was set to achieve a test power of .85 to 
reliably detect statistically significant effects, should they exist. The Type | error rate was set at .05. Based on these 
parameters, the power analysis indicated that a sample of at least 1050 teachers was required to reliably detect 
the anticipated effects. Based on this analysis, a decision was made to obtain a sample of 1215 teachers, which 
exceeds the minimum required sample size determined by the power analysis. Teachers were selected from differ- 
ent regions, including urban and rural areas, to ensure a diverse and representative sample. This stratified sampling 
approach aimed to capture the varied experiences and perspectives of teachers from different geographical and 
demographic backgrounds, allowing more robust generalization of the results. Although a non-probabilistic method 
was employed, teachers were randomly selected based on their accessibility and willingness to participate. This 
approach was chosen to maximize participation while ensuring a diverse range of respondents. The breakdown 
of participating teachers is as follows: 182 primary school teachers (teaching all school subjects), 268 social sci- 
ence teachers, 227 science teachers, and 246 vocational teachers. In terms of teaching experience, most of the 
participating teachers had more than 20 years of experience (50.04 %), followed by 15.97 % of teachers with 15-20 
years of experience, 13.00 % of teachers with 5 years of experience or less, 12.02 % of teachers who have been 
teaching for 10-15 years, and 8.97 % of teachers who reported 5-10 years of experience. The varied experience 
levels further enhance the representativeness of the sample, capturing insights from both seasoned educators 
and those newer to the profession. 


Instrument and Procedures 
The research was conducted at the end of the 2021/2022 school year, coinciding with the conclusion of emer- 
gency remote teaching. The primary objective was to gather insights into the pedagogical strategies employed 


by teachers during the remote teaching phase, to develop pedagogical recommendations and guidelines in case 
the need for remote teaching arises again. The initial questionnaire consisted of 18 items representing dependent 


854 


ASS https://doi.org/10.33225/jbse/23.22.851 


Journal of Baltic Science Education, Vol. 22, No. 5, 2023 


ISSN 1648-3898 /Print/ SCIENCE TEACHERS’ APPROACH TO CONTEMPORARY ASSESSMENT WITH A READING 
LITERAGY EMPHASIS 
ISSN 2538-71 38 /Online/ (PP. 851-864) 


variables and 5 items representing independent variables. To ensure the instrument's robustness, a validation study 
was initiated to scrutinize the questionnaire’s structure and affirm its construct validity. The study aimed to vali- 
date the measures used to assess teachers’ assessment in the time of emergency remote teaching. The questions 
were based on teachers’ experiences reported in various courses they attended during the pandemic. Participants 
included 2 teachers from the first triennium of primary school, 2 teachers from the second triennium of primary 
school, 2 teachers from the third triennium of primary school, 2 teachers from a vocational high school, and 2 
teachers from a grammar school. Based on the results, the questionnaire was found acceptable. 

The final questionnaire for teachers was designed and published on the online platform 1ka.si. It was distributed 
through various forums and websites to teachers who matched the predetermined sample. The questionnaire was 
also distributed among teachers with a request to share the link with their colleagues. For the original purpose of 
drawing conclusions about remote teaching, half of the results were used, except for the question about assessment 
during remote teaching. However, with the emergence of Al and its broader impact on education, the question 
of assessment in new learning environments and under new instructional and teaching conditions has become 
relevant again. To understand teachers’ readiness for unconventional assessment methods, answers to questions 
posed in these new learning environments can provide valuable insights. Due to teachers’ limited awareness of Al 
and the practical benefits it can have for them, as shown by a survey conducted in Estonia, which ranks first among 
the 27 European countries in the Index of Readiness for Digital Lifelong Learning (IRDLL) (Chounta et al., 2022), 
a new questionnaire was deliberately not designed, as the original one also covers new teaching methods, even 
in the age of Al (Zimmerman, 2018). This decision was made because a similar question related to the environ- 
ments with which teachers are already familiar after the COVID crisis provides sufficiently clear results and answers 
regarding teachers’ opinions about unconventional assessment methods in the classroom. These responses are 
later interpreted in the context of the emerging construct of Al. The responses to the questions asked in these new 
learning environments may help to understand teachers’ readiness for unconventional assessment methods. The 
surveyed question consisted of five items that provided data on assessment approaches during emergency remote 
teaching. Teachers answered these questions on a 4-point Likert scale (1 — never, 2 - rarely, 3 - often, 4 - mostly). 
They rated the frequency of assessment through videoconferencing, quizzes and tasks in online classrooms, written 
assessments, evaluation of seminar papers, and assessment of authentic assignments, video products, and projects 
during emergency remote teaching. Finally, teachers also responded to a demographic question, from which the 
information about their primary educational domain was obtained. The options for selection were primary school 
teacher, social sciences teacher, natural sciences teacher, and vocational subject teacher. 

The Cronbach's alpha coefficient, with a value of .771, showed reliable internal consistency (Nunnally, 1978). 
The commonality data indicated the extent to which each variable contributed to the extracted factors, and all 
variables had appropriate values for further interpretation (oral assessment using video conferencing = .568; quizzes 
and tasks in online classrooms = .523; written assessment = .567; evaluation of seminar papers = .553; assessment 
of authentic assignments, video products, and projects = .797). 


Ethical Procedures 


All measurements and interventions were conducted in accordance with applicable ethical guidelines and 
were voluntary for teachers who chose to participate in the study. Prior to their involvement, teachers provided 
informed consent, granting permission for the utilization of their data for analytical purposes and subsequent 
publication of findings. All information collected was processed and stored in accordance with applicable data 
protection regulations. A special emphasis was placed on safeguarding the privacy and ensuring the anonymity 
of the participating educators. To this end, all collected materials maintained teacher anonymity through the 
deployment of encrypted identification codes, eliminating any possibility of tracing back to individual identities. 
Throughout the entirety of the research process, the highest ethical standards were rigorously upheld. The rights, 
dignity, and overall well-being of all participants were consistently prioritized and respected. 


Data Analysis 
Reliability analysis was conducted to examine the questionnaire structure. To measure the internal consistency 


of the questionnaire, Cronbach's alpha coefficient was employed. The Kruskal-Wallis analysis for independent 
samples was performed to test for statistically significant differences among groups of teachers based on their 


855 


https://doi.org/10.33225/jbse/23.22.851 LTT 


Journal of Baltic Science Education, Vol. 22, No. 5, 2023 
SCIENCE TEACHERS’ APPROACH TO CONTEMPORARY ASSESSMENT WITH A READING ISSN 1648-3898 /Print/ 


Reoeheeo ISSN 2538-7138 /oniines 


primary educational domain. In the context of the Kruskal-Wallis test, the magnitude of the effect of the observed 
differences was calculated. Furthermore, a post hoc analysis using Dunn's test was conducted to explore specific 
differences between groups of teachers. Descriptive statistics were employed to obtain basic information on mean 
scores, standard deviations, and ranges within each assessment category. The data were analyzed using IBM SPSS 
software. 


Research Results 


In accordance with the research conducted, the general results regarding the forms of assessment during 
emergency remote teaching are presented, based on how frequently they were chosen by the teachers. 


Table 1 
Frequency of Using Different Assessment Methods during Emergency Remote Teaching 


Assessment Method N M SD 
Oral assessment using video conferencing 1215 2.39 1.176 
Quizzes, tasks in online classrooms 1215 2.07 1.038 
Written assessment 1215 1.54 869 
Assessment of seminar papers 1215 1.95 1.012 
Assessment of authentic tasks, video productions, projects 1215 2.29 1.121 


The results presented (Table 1) show that during emergency remote teaching, teachers predominantly assessed 
students’ knowledge through oral assessment via video conferencing (M = 2.39). The average of all responses given 
is also found for the frequency of assessment through authentic tasks, video productions, and projects (M = 2.29). 
Less frequently, teachers chose quizzes and tasks in online classes (M = 2.07) and assessment of seminar papers 
(M = 1.95) less frequently, while they least frequently chose to assess student knowledge through written assess- 
ment (M = 1.54). The variable of written assessment showed the least variability in teachers’ responses (SD = .869), 
indicating greater consensus among teachers regarding the choice of written assessment in distance learning. The 
greatest dispersion of results was found for oral assessment using video conferencing (SD = 1.176) and assessment 
of authentic tasks, video productions, and projects (SD = 1.121), suggesting that teachers vary in how well prepared 
they are to assess in this way. However, the variance is still relatively small, suggesting some degree of consistency 
in assessment using the above methods. Slightly, but not significantly, different scores were observed for quizzes 
and tasks in online classrooms (SD = 1.038) and assessment of seminar papers (SD = 1.012), where the teachers’ 
scores were somewhat more similar. 


Table 2 
Assessment Methods during Emergency Remote Teaching according to Their Primary Field of Teaching 


‘ , . P Vocational subject 
Primary school teacher Social sciences teacher Natural sciences teacher J 


teacher 
N M N M N M N M 
Oral 182 412.94 368 521.75 418 462.07 247 445.43 
Quizzes 182 425.81 368 438.42 418 510.22 247 507.21 
Written 182 364.60 368 465.78 418 487.32 247 515.57 
Seminar 182 295.25 368 472.69 418 465.79 247 591.93 
Authentic 182 406.52 368 484.33 418 441.34 247 507.60 


Note. Oral - Oral assessment using video conferencing. Quizzes - Quizzes, tasks in online classrooms. Written - Written assess- 
ment. Seminar - Assessment of seminar papers. Authentic - Assessment of authentic tasks, video productions, projects. 
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The results presented in Table 2 show statistically significant differences in the choice of assessment methods 
among teachers based on their teaching area (H(3) = 23.249, p =.001, n? =.02). Video conferencing assessment was 
most frequently used by social science teachers (M = 521.75), which is consistent with their preference for interactive 
teaching methods and discussions with students. Science teachers (M = 462.07) and vocational subject teachers 
(M = 445.43), on the other hand, used it less frequently, possibly due to the practical nature of their subjects, which 
is more difficult to assess through videoconferencing. Primary school teachers (M = 412.94) were the least likely to 
use it, likely due to the challenges of providing appropriate technology and addressing the developmental needs 
of younger students. The small effect size (n? = 0.02) does not detract from the statistical significance of the dif- 
ferences found (p = .001). The teaching domain is only one of potentially many influencing factors, which means 
that further research is needed. 

Based on the results of the Kruskal-Wallis test (H(3): 19.642, p = .001, n’ =.02), significant differences were found 
between at least two groups of teachers in their use of quizzes and assessment tasks. Teachers of science subjects 
(M = 510.22) and teachers of vocational subjects (M = 507.21) most frequently assessed their students using quizzes 
and tasks, which could be attributed to the interactive and practical nature of their subjects that lend themselves 
these assessment methods. Social science teachers (M = 438.42) and primary school teachers (M = 425.81) used 
quizzes and tasks less frequently, which may be due to the emphasis on conceptual understanding in social sci- 
ence subjects and the use of alternative assessment methods for primary school students to gauge comprehensive 
understanding. Despite the small effect size (n’ = 0.02), the statistical significance of the observed differences 
(p = .001) remains unchanged. The pedagogical domain is only one of many possible elements influencing these 
differences, highlighting the need for further research. 

Statistically significant differences in the frequency of written assessment (H(3): 45.005, p = .001, n’ = .04) 
were found between the different groups of teachers. Teachers of vocational subjects (M = 515.57) chose written 
assessment most frequently, probably because it is suitable for assessing skills and subject knowledge. Natural 
science teachers (M = 487.32) and social science teachers (M = 465.75) used written assessment less frequently 
because their subjects focus primarily on theoretical concepts that may be better assessed using other methods. 
Primary school teachers (M = 364.60) were least likely to use written assessment, possibly due to the challenges of 
developing writing and reading skills in younger students, leading them to choose more appropriate assessment 
approaches for their developmental stage. The observed effect size (n? = .04) means that although the differences 
are statistically significant, the variance in the frequency of writing assessments between the different teacher 
groups is only marginally explained by the instructional domain. This suggests that other factors may also play a 
significant role in teachers’ choice of assessment method. 

The analysis also revealed statistically significant differences in assessment method choices based on teach- 
ing domains (H(3) = 124.784, p = .001, n* = .10). Vocational subject teachers (M = 591.93) most frequently used 
seminar paper assessment, reflecting the specificity of vocational education, in which seminars are a common way 
of assessing practical skills and knowledge related to the chosen profession. Social science teachers (M = 472.69) 
and natural science teachers (M = 465.79) used this method less frequently, possibly reflecting the nature of 
their subjects, which require other appropriate assessment methods. Primary school teachers (M = 295.25) used 
seminar assessments least frequently, which may be related to the adaptation of assessment methods to the age 
group and developmental characteristics of younger students at this educational level. The relatively large effect 
size (n° = 0.10) in this analysis suggests that 10% of the variance in the frequency of seminar paper assessments 
among the different groups of teachers can be attributed to their teaching domain. This suggests that the teaching 
domain has a considerable influence on the choice of this assessment method. Despite the observed differences, 
factors other than the teaching area could also affect teachers’ assessment preferences. 

Similarly, significant differences were observed in the use of assessment of authentic tasks, video productions, 
and projects were observed (H(3): 17.503, p = .001, n? = .02). Vocational subject teachers (M = 507.60) frequently 
emphasized practical skills, best assessed through authentic tasks, video productions, and projects. Social science 
teachers (M = 484.33) also frequently used this method, possibly to encourage critical thinking and idea expres- 
sion. Natural science teachers (M = 441.34) used it less frequently, possibly due to a greater emphasis on concrete 
scientific concepts. Primary school teachers (M = 406.52) used this assessment method least often, likely due to 
the constraints of conducting such assessment formats with younger students. The effect size suggests that the 
teaching domain accounts for 2 % of the variance in the use of authentic tasks, video productions, and projects for 
assessment, suggesting that other factors contribute significantly to this choice and thus further research is needed. 

A Dunn's post-hoc test aimed to compare the assessment practices of natural science with those from other 
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fields in new educational environments. The results suggest that natural science teachers’ assessment methods 
in these contexts do not significantly differ from other teachers. However, some notable differences emerged in 
specific assessment types. Regarding oral assessment, a statistically significant difference was found between 
natural science teachers and social science teachers (p = .008, M — =2.35,M_. = 2.62). Natural science 
natural science social science 
teachers tended to use oral assessment less frequently compared to their counterparts in social science. In the case 
of quizzes, significant differences were observed between natural science teachers and primary school teachers 
(p =.009, M _ =2.26,M_. = 1.91) as well as social science teachers (p = .007,M — =2.26,M__ 
natural science primary school natural science social science 
= 1.96). Regarding written assessment, statistically significant differences were seen in comparison with primary 
school teachers (p = .001, M  =1.63,M_. = 1.17). Natural science teacher teachers tended to use writ- 
natural science primary school 
ten assessments more frequently than primary school teachers. For the assessment of seminar papers, differences 
were found between natural science teachers and primary school teachers (p = .001,M . =1.94,M_. = 
natural science primary school 
1.29), as well as vocational subject teachers (p = .001,M — =1.94,M_. = 1.24). Natural science teachers 
natural science primary school 
used assessment of seminar papers more frequently compared to primary school and vocational subject teachers. 
However, no statistically significant differences were observed in the use of authentic tasks among natural science 
teachers and teachers from other fields. In summary, while natural science teachers did not significantly differ from 
teachers in other fields in terms of general assessment practices in new environments, they showed variations in 
specific assessment methods like oral assessments, quizzes, written assessments, and assessment of seminar papers 


when compared to their colleagues in primary school and social science, as well as vocational subject teachers. 
Discussion 


The purpose of this study is to examine science teachers’ preparedness for assessment in the age of Al, con- 
sidering the challenges and opportunities presented by the ubiquitous presence of Al technologies in education 
(Cotton et al., 2023). Although the results were analyzed from the perspective of self-assessment during periods of 
emergency remote teaching, the findings suggest that factors beyond the teaching domain also play significant 
roles in determining teachers’ choice of assessment methods. These conclusions are further discussed through 
the lens of assessment in the age of Al. 

Based on the theoretical framework of the research conducted, it is hypothesized that teachers, in general, 
are not adequately prepared for assessment in the era of widespread accessibility of Al. In line with the theoretical 
underpinnings of the study, which highlights the need for new forms of knowledge, new forms of learning, new 
forms of teaching, and consequently new forms of assessment, it was expected that teachers would predominantly 
report choosing conventional assessment methods during emergency remote teaching. This period opened the 
door to computer-based learning and new forms of assessment in schools (Khan et al., 2023). Results show that 
teachers most often chose to use oral assessment via video conferencing during distance assessment. This method 
was instrumental in curbing potential plagiarism and other forms of cheating that students might employ in 
assessing their curriculum goals. Oral assessment is considered one of the fundamental forms of conventional 
assessment, alongside with written assessment, which was used least frequently during remote teaching due to 
concerns of plagiarism and cheating. Oral assessments can be successful and necessary in contemporary learning 
environments when they promote self-reflection, performance-based assessment, and peer assessment (Crawford 
et al., 2023; Gimpel et al., 2023). To a slightly lesser extent, but still infrequently, teachers chose to assess authentic 
tasks, video productions, and projects, which are forms of assessment anticipated to be foundational in future 
assessments. Teachers were somewhat less inclined to assess seminar assignments, which became problematic in 
the age of Al because students could complete the entire assignment using Al. Teachers’ responses indicate that 
they are not fully prepared for assessment in the age of widespread Al in education, underscoring the need for 
enhanced teacher preparation for Al use in assessment. Adequate training and support for teachers are essential 
to help them navigate the challenges and harness the opportunities that Al technology presents in education, 
as it becomes evident that they are not yet ready for it (Kerneza et al., 2023). Adapting educational practices and 
policies to the evolving technological landscape of education triggered by distance learning (Gurajena et al., 2021) 
is crucial for achieving teacher readiness for assessment in the age of Al. 

The results indicate that differences exist in the assessment approaches adopted by teachers during remote 
education, contingent on their primary area of instruction. Oral assessment via videoconferencing was most fre- 
quently chosen by social science teachers, less so by natural science and vocational teachers, and least by primary 
school teachers. The predilection for more interaction and discussion in social science subjects might explain this 
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trend, while natural science subjects entail more hands-on demonstrations or experiments, challenging to execute 
in a virtual milieu. Quizzes and assignments were predominantly utilized by natural science and vocational teach- 
ers to assess student knowledge, and least by social science and primary school teachers. Natural science and 
vocational teachers appear to emphasize the practical application of knowledge, assessable through assignments 
and quizzes, whereas social science and primary school teachers seem to prioritize conceptual understanding, 
assessable through alternative assessment forms. Written assessments were predominantly chosen by vocational 
teachers, less so by natural science and social science teachers, and least by primary school teachers. The variance 
in the preference for written assessment among different subject teachers could stem from the intrinsic nature of 
the subject and the imperative for written articulation of knowledge. Vocational subjects might necessitate ex- 
tensive writing and formulation, aligning with conventional assessment methods (Montgomery, 2022) pertinent 
to skills typically not computer based (Swiecky et al., 2022). In contrast, social science subjects might emphasize 
argumentation and essay writing. Science problems are more easily assessed using alternative methods, potentially 
explaining why natural science teachers infrequently opt for written assessment. Assessment of seminar papers 
was most prevalent in vocational subjects, less so in social studies and natural science subjects, and least in primary 
school subjects. Vocational students might develop seminar papers emphasizing practical examples and real-world 
knowledge applications, while social science subjects might focus on data presentation, and natural science sub- 
jects on experimental work. Conversely, primary school students, especially in the early years, might not possess 
the requisite skills for independent seminar paper production. Authentic tasks, video productions, and projects 
were predominantly chosen by vocational and social science teachers, less frequently so by natural science teach- 
ers, and least by primary school teachers. This aligns with the practical and analytical skills students demonstrate 
in such assignments. Natural science subjects might necessitate experimental work or the creation/presentation 
of a scientific research project. Science teaching involves intricate concepts that students encounter and explore 
in school, with optimal teaching and learning defined by higher taxonomic levels (CAST, 2018). Primary school 
teachers, who also teach science, were observed to seldom use authentic tasks, video productions, and projects. 
Given the younger age of these students, it is not anticipated that these tasks and related skills are as developed as 
in older students. However, it becomes imperative to ascertain whether teachers are fostering basic digital literacy 
skills in students, preparing them for advanced assessment at the secondary level. Primary education is a pivotal 
juncture in establishing the foundation for teaching contemporary 21st-century learning environments (Kerneza 
& Kordigel Abersek, 2023; Kordigel Abersek & Kerneza, 2023). 

Considering the research findings that address potential disparities between science teachers and educators 
from other disciplines concerning assessment practices in the age of pervasive Al accessibility and the unparal- 
leled innovations in science and technology, as highlighted by OECD (2018), some differences were noted in the 
selection of assessment methods based on the educators’ primary instructional domain. Social science teachers 
predominantly opted for oral assessments via videoconferencing, suggesting an amplified necessity for interac- 
tion and discourse within social science disciplines. In contrast, science and vocational subject teachers frequently 
employed quizzes and assignments as assessment tools, potentially reflecting the accentuation of knowledge's 
practical application within this domain. Vocational teachers exhibited a marked preference for written assessments, 
whereas their counterparts in the social sciences appeared to prioritize argumentative and essay-based evaluation. 
Science teachers demonstrated a reduced inclination towards written assessment, possibly due to the efficacy of 
alternative methods in evaluating scientific queries. The assessment of seminar papers was predominantly observed 
within the vocational subjects, facilitating students in showcasing practical knowledge applications in real-world 
context. Primary school teachers exhibited a reduced propensity for this assessment form, attributing this to the 
perceived underdevelopment of independent seminar paper production skills among primary students. Authentic 
tasks, video productions, and projects were predominantly favored by vocational and social science educators, 
mirroring the practical and analytical proficiencies students manifest in these tasks. Science teachers displayed a 
diminished preference for these assessment methods, potentially due to the inherent requirements of experimental 
undertakings or the formulation of scientific research projects, which might be deemed unsuitable for primary 
school students lacking the requisite competencies for autonomous task execution. 

The findings from the research highlight the varying degrees of teacher readiness and their inclinations 
towards certain assessment methods during remote teaching. These can be interpreted through the lens of 
potential receptiveness to Al-driven innovations. Such receptiveness is not merely a reflection of adaptability to 
new technologies but also an indication of pedagogical flexibility and willingness to evolve in response to the 
changing educational landscape. Insights from the research phase suggest that the experiences and feedback of 
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teachers during distance education can provide valuable context when considering the broader implications of 
Al-driven assessment. For instance, the challenges faced, the solutions devised, and the overall sentiment towards 
technology-mediated teaching can serve as indicators of how educators might approach and integrate Al tools in 
their teaching and assessment practices. Moreover, the nuances of experiences during remote teaching, such as 
preferences for certain assessment methods or reservations about others, can offer clues about potential areas of 
comfort or concern when it comes to Al-driven assessment. This perspective is further supported by recent research, 
including the works of Salas-Pilco, Xiao, and Hu (2022). Their findings emphasize the multifaceted nature of teacher 
readiness, suggesting that it's not just about technological proficiency but also about pedagogical understanding, 
attitude towards innovation, and the ability to foresee the potential challenges and benefits of integrating Al. Col- 
lectively, these insights paint a comprehensive picture of the evolving landscape of Al in education, emphasizing 
the intricate interplay between technology, pedagogy, and the requisite preparedness of educators. 

The data highlight the imperative for literacy skills that are essential for participation in various assessment 
activities, encompassing oral evaluations, reading assignments, essay writing, or comprehension of instructions. 
Teachers must recognize the significance of fostering students’ literacy skills and catering to their diverse reading 
and writing requirements within the assessment context. Such an approach not only renders learning meaningful 
but also proffers students with significant and rigorous learning prospects, as underscored by CAST (2018). This is 
especially pronounced in primary schools, where the same teacher assesses both science and literacy, given the 
low reading scores shown in the new PIRLS data analysis (Mullis et al., 2023). This ensures equitable and quality 
knowledge assessment in the Al epoch, wherein the widespread availability of Al technologies in education repre- 
sents merely one facet of the myriad opportunities and challenges confronting teachers. The connection between 
literacy and science instruction is crucial, as evidenced by other studies (e.g., Kim et al., 2021; Pearson et al., 2010). 
Grasping content and critical interpreting of scientific texts are vital for students’ academic success in science, 
given their exposure to a plethora of scientific texts that often contain complex concepts, scientific language, and 
specific terminology. Proficiently navigating scientific literature is paramount for information assimilation, key idea 
identification, detail discernment, and holistic subject comprehension. Within contemporary assessment frame- 
works, students must be able to evaluate the credibility of sources, recognize scientific bias, analyze, and evaluate 
arguments, and identify possible errors or gaps in research. This requires developed critical thinking supported 
by advanced reading and writing skills. Following the process of data collection, students should articulate their 
perspectives coherently and systematically, defending their viewpoints and discoveries. Written expression skills 
include appropriate use of scientific language, text organization, logical connections, and use of appropriate scien- 
tific and technical terms. In summary, the teaching and assessment of science in the age of Al encompasses the full 
spectrum of both literacy and scientific content. In a modern science classroom supported by Al, the development 
and support of literacy skills are crucial. Students must have the opportunity to develop comprehension, critical 
reading, summarizing, and analytical thinking skills in the specific context of science content. This prepares them 
with the skills essential for active participation in assessment activities demanding the interpretation and applica- 
tion of scientific data. Students should instruct students to critically evaluate sources and information they obtain 
from the Internet, which should also be given more attention by teachers (Zemljak & Kerneza, 2023). A potential 
strategy is delineated by Leu et al., (2008) with the three-phase model of online reading instruction called Internet 
Reciprocal Teaching (IRT), resonating with the tenets of problem-based learning (Zemljak et al., 2023). This approach 
facilitates student collaboration, fostering critical thinking and problem-solving skills, with an emphasis on the 
learning trajectory rather than mere outcomes. Students need empowerment within the learning paradigm, given 
that the learning process itself constitutes a pivotal contribution to pragmatic life knowledge- Thus, education 
can be perceived as a life experience, encompassing attitudes, knowledge, and skills, tailored for life-encompassing 
behavior, cognition, and consideration (Broks, 2023). 

Children today live in a very different world than their parents (Siraj, 2017). Not only do today’s children 
not know the time before smart devices; they are also the first generation whose lives are in one way or another 
defined by Al-enabled applications and devices, while also being exposed to Al-related risks (UNICEF, 2020). The 
research provides insights into the need for training and support for teachers in the use of Al in assessment, but 
further study is needed to provide a more concrete and reliable picture of teacher readiness in this area. If Turetzky 
et al., (2019) were already calling for Al researchers to become Al educators in 2019 and create resources to help 
teachers and students understand Al intelligence, this is even more important in 2023. Kerneza (2023) noted that 
pre-service teachers frequently overestimated the competencies essential for interpreting chatbot-generated con- 
tent, a perception that often diverged from evaluators’ assessments. Such disparities might have led to potential 
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misjudgments of their capabilities or suboptimal evaluations, potentially curtailing their progression opportunities. 
This reinforces the findings outlined by Farell (2007) that teachers need to continually revise their knowledge of 
teaching and learning to teach effectively and competently in the rapidly changing field of education. With addi- 
tional research and analysis focused directly on Al's role in assessment, more definitive conclusions can be drawn 
regarding teacher readiness in this sphere, facilitating the proposition of suitable strategies and interventions to 
recalibrate educational practices. Our responsibility as individuals is to ensure the ethical development and ap- 
plication of Al (Dignum, 2019). Therefore, as UNESCO (Miao et al., 2020) emphasizes it is particularly important, 
that Al supports and promotes children’s growth. More than ever, education must become more systemic through 
a balance between technological and humanitarian education. Today, we need to find better ways to combine 
our physical and spiritual pursuits, which is the fulfillment of a truly sustainable (balanced, long-term) develop- 
ment of our way of life and our future education (Broks, 2016). Furthermore, it is important to emphasize that, 
as UNESCO (Miao et al., 2020, p. 39) point out, the teacher must support children’s development and well-being; 
ensure inclusion of and for children; prioritize fairness and nondiscrimination for children; protect children’s data 
and privacy; ensure children’s safety; provide transparency, explainability, and accountability for children; prepare 
children for current and future developments in Al; equip governments and businesses with knowledge about Al 
and children’s rights; and create an enabling environment for child-centred Al. This can only be achieved through 
an Al-competent teacher. 

The uniqueness of this study lies in its emphasis on the readiness of science teachers for assessment in the age 
of Al. While there have been studies on the use of Al in education and its impact on student learning, this research 
specifically addresses teachers’ readiness to use Al-based assessment methods. By examining teachers’ perspectives 
and practices, it sheds light on the current state of their readiness and highlights areas that need further attention 
and support. This focus on teacher readiness in the context of Al assessment sets this study apart from others. The 
implications of the study also extend to the broader educational community, highlighting the need for professional 
development programs, subject-specific adaptations of Al-based assessment, and the integration of literacy into 
assessment practices. By considering these implications, the findings can contribute to the effective and ethical 
implementation of Al in education, ultimately benefiting student learning outcomes. 

The limitations of the study are also considered in the analysis of the results and discussion. A non-probability 
sample was used for the study, which may lead to selective participation and potential bias, as only those teachers 
who are more interested in the topic or have experience may have chosen to participate. Therefore, the generaliz- 
ability of the results should be taken with caution, as they may be more representative of this group of teachers. With 
respect to the administration of the questionnaire, it is also important to consider the possibility that teachers may 
have provided subjective responses that are consistent with socially desirable responses or idealized perceptions 
of their work. There are also concerns about the accuracy of self-reflection. In addition, it is important to note that 
the study was conducted during a time when teaching occurred remotely, which may have influenced teachers’ 
perceptions, instructional decisions, and willingness to adopt new assessment methods. Therefore, the application 
of the study’s findings under “normal” circumstances should be carefully considered. 


Conclusions and Implications 


This research highlights the need for better preparation and support of teachers in the use of Alin assessment. 
Despite the challenges and opportunities Al presents in education, teachers predominantly chose conventional 
assessment methods during emergency remote teaching. This approach helped address concerns about plagia- 
rism and cheating. However, the assessment methods teachers chose varied depending on their primary subject. 
Certain assessment methods were preferred across subjects, such as quizzes and assignments in natural science 
and vocational subjects and written assessments in vocational subjects. Primary school teachers were less likely 
to use certain forms of assessment because of the age and readiness of their students. The study also highlights 
the importance of literacy skills and their connection to science education. 

The findings suggest that teachers are not fully prepared for assessment in the era of widespread availability 
of Al technologies. Understanding teacher readiness for assessment using Al is critical to the successful adaptation 
of this technology in education. Only with adequately trained teachers can we ensure that Al serves as a tool to 
enhance the learning process and not as a substitute for teacher expertise. Professional development programs, 
subject-specific adaptations of Al-based assessment, and the integration of literacy into assessment practices are 
essential to ensure the effective implementation of Al in education. To further this goal, it is recommended that 
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future efforts be directed towards the development of comprehensive educational programs and materials. For 
the organization of the educational process, it is recommended that ongoing professional development programs 
focused on the use of artificial intelligence in assessment are introduced. Additionally, it is suggested that additional 
resources and tools are provided for teachers to assist them in adapting to new technologies and assessment meth- 
ods. In the context for planning further research, it is proposed that studies be conducted to directly assess specific 
teacher skills reflecting their readiness to work in the reality of artificial intelligence. Such studies would allow for 
a more precise understanding of where the exact shortcomings lie and how best to address them. This will ensure 
that teachers are not only introduced to Al but are also equipped to utilize it effectively in their teaching practices. 

As we stand at the brink of an educational shift powered by artificial intelligence, the readiness of our teach- 
ers is the key to its success. The interplay between traditional teaching methods and emerging Al technologies is 
intricate, and it’s up to us to ensure this collaboration is seamless. The future of education, shaped by Al, will be as 
strong as the foundation we lay today. Thus, investing in our teachers’ preparedness is not just a necessity but a 
commitment to a brighter, more informed future. 
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Introduction 


Human activities such as deforestation, overfishing, and environmental 
pollution have endangered the survival of many species; for example, com- 
mercial whaling and marine garbage have brought whales close to extinction. 
To achieve a harmonious co-existence of humanity and nature, the United Na- 
tions has launched a program titled ‘Education for Sustainable Development’ 
Accordingly, understanding of ecological issues, such as climate change and 
biodiversity loss has become particularly stressed in environmental education 
and sustainable development (Mambrey et al., 2022), and the ecosystem has 
emerged as an important aspect in international science curricula (Australian 
Curriculum, Assessment and Reporting Authority, n.d.; Ministry of Education, 
&P.R. China, 2022; Next Generation Science Standards Lead States, 2013). 

Ecological issues refer to environmental problems concerning organ- 
isms, and their interactions with each other and the environment (Ecology 
Center, 2023). These are complex systems that are difficult to understand, 
because of the inherent intricacies, and long-term effects of human actions 
(Fanta et al., 2020). Multiple dimensions (e.g., social, economic, and socio- 
cultural elements) must be considered for identifying the consequences of 
such issues (Fanta et al., 2020; Ke et al., 2020; Riess & Mischo, 2010; Schuler 
et al., 2018). However, students lacking a systematic perspective cannot fully 
appreciate these issues (Ke et al., 2020; Liu et al., 2011; Yang, 2005). To solve 
these issues effectively, students need the systems thinking (ST) skills (Hogan, 
2002; Mambrey et al., 2022; Riess & Mischo, 2010) that are emphasized in 
international science education documents (e.g., National Research Council, 
2010; Next Generation Science Standards Lead States, 2013). ST is the skill to 
reason about complex systems in accordance with system characteristics to 
create a holistic understanding of various phenomena for problem-solving 
and system evaluation (Fanta et al., 2020; Gilissen et al., 2020; NRC, 2010). 
Researchers in various fields advocate the necessity to promote students’ ST 
in science learning (Gilissen et al., 2020; Schuler et al., 2018; Sommer & Licken, 
2010; Verhoeff et al., 2018). Moreover, complex and diverse perspectives of 
ecological issues would allow students to engage in ST about broader issues, 
and ultimately achieve global citizenship literacy (Ke et al., 2020; Lee et al., 
2013; Sadler et al., 2007). Thus, it is essential to explore and develop students’ 
ST in the context of ecological issues. 
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Abstract. Exploring students’ systems 
thinking (ST) is essential in enhancing 
science learning, but existing studies have 
failed to understand students’ ST fully as 
it relates to ecological issues. This study 
aimed to fill the aforementioned literature 
gap by exploring lower-secondary school 
students’ ST regarding ecological issues. 
The Systems Thinking Test regarding 
Ecological Issues, which measures four 

ST skills (system organisation, behaviour, 
application, and evaluation), was 
administered to 1,092 lower-secondary 
school students. The results reveal low ST 
performance in ecological issues, with 
students finding it particularly difficult to 
identify interactions among components 
and understand system characteristics. 
Furthermore, most lacked reflective 
consciousness and consideration of the 
diverse dimensions of ecological issues, 
resulting in monocausal reasoning in 
system decision-making and evaluation. 
Comparatively, urban school students 
performed better than their rural 
counterparts; additionally, an item-level 
analysis revealed that climate warming 
was challenging for the students to 
understand. This study suggests that 
greater efforts should be made to address 
students’ drawbacks and that multi- 
perspectival teaching is necessary in the 
context of ecological issues. The addition 
of system decision-making and evaluation 
in assessments can enable a broader 
understanding of ST. 
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ST in the Ecological Context 


According to previous studies, ST in science education incorporates not only the skill to identify interactions 
among system components and understand system characteristics (Ben-Zvi Assaraf & Orion, 2005; Brandstadter et 
al., 2012; Mehren et al., 2018; Sommer & Liicken, 2010) but also the skill to make predictions, decisions, and judge- 
ment based on system analysis (Ben-Zvi Assaraf & Orion, 2005; Evagorou et al., 2009; Mehren et al., 2018; Riess & 
Mischo, 2010) and make evaluations on the validity of system structures, decision-making, and the limitations of 
predictions (Fanta et al., 2020; National Research Council, 2010; Rosenkrdnzer et al., 2017). 

Given that multiple components and interactions of ecosystems must be considered, studies have mostly 
focused on students’ understanding of interdependent relations in ecosystems; for example, assessments of elemen- 
tary school students require them to reason about food webs (Hokayem & Gotwals, 2016; Mambrey et al., 2020; 
Mambrey et al., 2022). At the secondary level, they focus on students’ scientific explanations of ST concepts regard- 
ing ecosystems (e.g., indirect connections, feedback loops, and emergent properties) (Jin et al., 2019). However, 
these studies failed to cover the full range of ST skills needed by students for understanding and solving complex 
ecological issues, especially for making decisions and evaluating on the basis of system analysis. Although the 
direct impact of human actions on ecosystems has been incorporated in the assessment (Jin et al., 2019), it failed 
to provide opportunities for students to consider the other dimensions (e.g., society, economy, and socio-culture) 
indirect effects on causing or solving ecological problems. 


Research on Lower-secondary School Students’ ST Levels 


Most studies (e.g., Hokayem & Gotwals, 2016; Jin et al., 2019; Mehren et al., 2018) have assumed that there is 
a gradual increase in the development of students’ ST skills through learning progressions (LPs). Some researchers 
have defined ST using a three-level hierarchical model (from low to high), namely, analysis of system components, 
synthesis of system components, and problem-solving (Ben-Zvi Assaraf & Orion, 2010). Jin et al. (2019) defined 
a four-level LP for ST in an ecosystem based on students’ response complexity patterns; the four levels are called 
no ideas (Level 1), individual organisms (Level 2), relationships and patterns (Level 3), and mechanisms (Level 4). 
Mehren et al. (2018) differentiated systems-thinking levels according to system complexity, including the number 
of components and interrelations, type of relations, and dealing with system properties. 

Only a few ST measurements at the lower-secondary school level exist, especially for large groups of students 
(Brandstadter et al., 2012). Existing studies indicate a low starting point for ST among young students with regard 
to various fields. A small sample study of the hydro-cycle system indicated a relatively low performance of lower- 
secondary school students regarding ST skills in general (Ben-Zvi Assaraf & Orion, 2005). A small-scale comparison 
of the skill of understanding complex systems between experts and Grade 7 students using the Structure—Behav- 
iour-Function model implied that novice students focused on the structural level (Hmelo-Silver & Pfeffer, 2004). An 
assessment of ST regarding ecosystems has revealed that most secondary school students tend to identify direct 
and indirect relationships (Jin et al., 2019). However, the specific difficulties hindering lower-secondary school 
students’ ST in the context of ecological issues remain unclear. 

Studies have also revealed a great improvement in ST performance following the introduction of teaching in- 
terventions at the lower-secondary school educational level (Ben-Zvi Assaraf & Orion, 2005; Hmelo-Silver et al., 2017; 
Riess & Mischo, 2010); this indicates the feasibility and potential of developing lower-secondary school students’ ST. 


Students’ ST Performance in Different Groups 


For a comprehensive evaluation of ST, it is important to detect differences in students’ average performance 
among different subgroups. However, this question has not been sufficiently explored in ST research (Cox et al., 
2019; Jin et al., 2019). 

Gender science stereotypes have long been discussed; females are still considered to be underrepresented 
in different science fields (Brotman & Moore, 2008). However, the effect of gender on ST is unclear; research in dif- 
ferent scientific contexts has yielded distinct results. A study conducted in engineering found that male students 
performed significantly better than female ones (Sweeney & Sterman, 2000). However, research on ecosystems and 
geographical systems has suggested no significant differences based on gender (Cox et al., 2019; Jin et al., 2019). 
Mambrey et al. (2020) found a similar reasoning pattern in elementary and lower-secondary school students, sug- 
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gesting that the ST of young students regarding ecosystems was independent of their age. However, a significant 
difference was found with respect to geographical ST between grades, especially among high school students 
(Cox et al., 2019). Research has found significant performance gaps in ST among schools from different locations 
in the United States (Jin et al., 2019). Because the above-mentioned studies have shown distinct results in different 
system-specific contexts, it is necessary to check the effects of the aforementioned variables on students’ ST perfor- 
mance in the more complex context of ecological issues involving interactions between humans and ecosystems. 


Research Aim and Questions 


A full picture of students’ ST regarding ecological issues is essential for promoting ST to achieve sustainable 
development. However, the existing studies failed to cover the full range of ST skills and the specific weaknesses 
that hinder students’ development of ST. Moreover, it remains unclear in the literature whether students are able 
to contemplate the various dimensions involved in these complex issues. Thus, this study aimed to supplement 
the missing content by comprehensively evaluating on a large scale lower-secondary school students’ST regarding 
ecological issues. The following were the research questions for this study: 

1. What is the current level of ST regarding ecological issues among lower-secondary school students? 
2. Inwhich specific aspects of ST regarding ecological issues do students experience the most problems? 
3. How do students from different groups differ in their ST performance? 


Research Methodology 
Description of the Test to Measure Students’ ST 


The authors constructed a systems thinking test regarding ecological issues (STTEI) based on Mehren et al’s 
(2018) ST framework; it incorporated comprehensive ST skills based on the latest insights from ST studies and 
was empirically validated in the context of geography and ecology (Mambrey et al., 2020). The items of the STTEI 
measured four skills in ST, which we regard as covering the full range of skills included in the existing literature 
(Ben-Zvi Assaraf & Orion, 2005; Brandstadter et al., 2012; Evagorou et al., 2009; Fanta et al., 2020; Mehren et al., 
2018; National Research Council, 2010; Riess & Mischo, 2010; Sommer & Lucken, 2010), namely, system organisa- 
tion, system behaviour, system application, and system evaluation. Items in system organisation refer to identifying 
ecosystem components and the interrelations among components. Items related to system behaviour require 
students to analyse the results of system changes based on the interactions involved and their understanding of 
the system characteristics. To apply the system, students were asked to make predictions and decisions based on 
a system-effect analysis. For the last skill, students were required to work on problems and determine the validity 
of the system structure, decisions made, and limitations of predictions. Second, these ST skills were differentiated 
into three levels based on performance complexity according to system-related structural and behavioural com- 
plexity, as described by Mehren et al. (2018). These levels are described below. 


Level 1: Students identify low numbers of elements and isolated or monocausal relationships. Students’ 
analysis and reasoning regarding systems are based on monocausal relations. 


Level 2: Students identify a moderate number of elements and linear-indirect relationships. The analysis 
and reasoning are based on linear and indirect relationships. 


Level 3: Students identify a large number of elements and complex or nonlinear relations. Students per- 
form analysis and reasoning based on nonlinear and complex relations. 


The STTEI comprised 13 items (ordered multiple choice, short answers, two-tied items, and extended response 
questions) to probe different levels of students’ ST about ecological issues, and each item was allocated to one skill 
(Table 1). As climate change and sustainability problems have been the most prevalent ecological issues in the 
last decade (Fanta et al., 2020; You et al., 2018), the test was about issues highlighting sustainability as the context 
that could provide students with opportunities to explore multiple dimensions in ecological issues (e.g. climate 
warming, sustainable rainforest development, overfishing, industrial pollution, abuse of pesticides, and the absence 
of top predators). Items were scored using 0-2 or 0-3 point ranges according to the three levels of performance 
complexity. The two tied items and extended response questions gave students the opportunity to express their 
explanations of ecological issues. The concept map (CM) item is an example of an extended response question 
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(Figure 1). It was designed based on the context of climate warming to assess the extent to which students identify 
interrelations among components related to climate warming. The provided concepts included both biotic and 
abiotic components. The social element (industrial production) was included in this item. The other items of the 
STTEI are presented in the Appendix. 


Figure 1 
Concept Map Item (Item 5) 


Climate warming has influenced human life a lot. The following picture provides 
seven concepts interrelated with climate warming. Use the given concepts and think 
about how they relate to each other. Connect them with arrows accordingly and 
remember to label each arrow (see example). Make sure to indicate clearly the 
direction of the arrows. 


Carbon Dioxide 


Oxygen 


Animals and Human 


Industrial production 


Plants 


Petroleum, Coal 


and Mineral 


To ensure content validity, the item content and wording were evaluated and revised by expert review. A 
quantitative pilot test (n = 97) was conducted to validate the test instrument, and the Rasch partial credit model 
(PCM) was used to examine item fit statistics to address construct validity. Three items in the original version were 
eliminated because of low item discrimination (below .2). Subsequently, a field test (n = 456) was conducted to 
validate further the 13-item test. The expected a posteriori (EAP) reliability was .8, indicating acceptable internal 
consistency, and all items’ mean-squares fell into the suitable range of .7 to 1.3 (Bond & Fox, 2007). 


Table 1 
Specific Information of Test Items 


Skills required Item context Item number 
Identify system components Pond ecosystem Item 1 
System organisation 
Identify interrelations among components Climate warming Item 5 
Understand emergence Ecological equilibrium Item 3 
Dynamic change Absence of top predators Item 8 
System behaviour 
Identify feedback loop Temperature change in forests Item 11 
Cyclic thinking Nutrition cycle Item 10 
Make predictions Industrial pollutant, Item 2 
Climate warming Item 6 
System application 
Make decisions Climate warming, Item 7 
Sustainable development of rainforests Item 12 
Evaluate the system structure Overuse of pesticides Item 4 
System evaluation Determine the validity of the system structure, deci- Absence of top predators Item 9 
sions, and limitations of predictions Overfishing problems Item 13 
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Participants and Data Collection 


Asample representative of the diversity of lower-secondary school students is required to obtain more robust 
statistical results and meet the purpose of the study, such as the requirement of comparing students’ performance 
across genders, grades, and school locations. Thus, six lower-secondary schools in southern and northern regions 
of China were purposefully sampled. These schools varied in school settings, educational resources, and socio- 
economic status. According to the previous study, the sample size is determined by the standard error of measures, 
which should exceed the minimal requirement of 267 (Liu, 2020; Wu et al., 2016). In order to improve the accuracy 
of measurement, the sample size is four times larger than the required minimal sample size (the standard error 
of Rasch measures less than .1). All participants in these schools possessed knowledge about the ecosystem; a 
total of 1,183 students from Grades 7 and 8 involved in the study. After eliminating blank and invalid responses, 
information from 1,092 students was used for the analysis (Table 2). 

Data were collected from November 2022 to December 2022. The purpose of the study and test content were 
reported to school principals to acquire permission. All students and teachers participated in the study volun- 
tarily, and the test was conducted anonymously to protect students’ privacy. The teachers ensured that students 
completed the test within 45 minutes. Before the test, teachers informed students of the aim of the test and that 
the results would be used exclusively for scientific research purposes. The teachers also promised students that 
their test performance would not affect their subject grades and encouraged them to write down their honest 
thoughts as much as possible. 


Table 2 
Demographic Data of the Participants 


Demographic categories Number of students Percentage (%) 
Ss oAiinealion Urban schools 592 54.2 
Rural schools 500 45.7 
eee Male 565 51.7 
Female 527 48.2 
Gis Grade 7 568 52 
Grade 8 524 47.9 


Data Scoring and Analysis 


The developed scoring rubrics were used to score all students’ responses. Ten percent of the randomly selected 
tests were scored by two independent raters (a PhD student and a lower-secondary school biology teacher) to check 
inter-rater reliability. An inter-rater reliability analysis using kappa statistics determined the consistency between 
raters, and the kappa’s range was .62-.85, with a median of .75 (Cohen, 1968), indicating substantial agreement 
for each item from different raters (Landis & Koch, 1977). 

This study used PCM analysis via the Test Analysis Module package in R software (Robitzsch et al., 2021) to 
establish the participants’ ST levels on the test, as it has been deemed suitable for analysing polytomous data. The 
reliability of the EAP estimates was checked, and the weighted likelihood estimates (WLEs) were used as student 
proficiencies. A Wright (person-item) map was generated to quantify the locations of item difficulty and student 
performance on the same logit scale. The t-test was used to compare performances between the different sub- 
groups of participants (e.g., male vs. female). Differential item functioning (DIF) was applied to examine differences 
between student groups at the item level to provide further information on weaknesses and strengths with regard 
to students’ ST (Wu et al., 2016). 
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Research Results 
Students’ Overall Performance in ST Regarding Ecological Issues 


In Figure 2, the left side of the Wright map displays the density distribution (which approximates a normal 
distribution) of the student performance (WLEs) on the test. The right side represents the distribution of the 
Thurstonian thresholds for each item. A comparison of the WLEs and item thresholds showed that only a small 
proportion of students had more than a 50% chance of achieving the third level (Figure 2). 


Figure 2 
Wright Map for ST Regarding Ecological Issues 
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Items 
Note. 1 = threshold value 1, 2 = threshold value 2, and 3 = threshold value 3. 


The threshold values of each item level were calculated using PCM analysis (Table 3), and the mean threshold 
values of the item responses belonging to the same level across the ST skills were calculated (Wang & Song, 2021). 
The mean of the students’ ability measures (WLEs) was compared with the mean threshold values of the different 
levels; when the mean WLEs exceeded the latter, it was determined that the student had reached a specific ST 
level. As shown in Table 3, the student ability measures reached, on average, Level 1 for each skill, and Level 3 was 
the most difficult to achieve for each skill. The students’ mean ability (M = .047) for system organisation was higher 
than that for the other three skills. 
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Mean Threshold Values of Every Performance Level across ST 


; ; mean Participant 
Skill Level Items and threshold values on different levels Threshold ability 
values 
Levelt I1(-1.034); 15(-.341) -.688 
System organisation Level2 11(-.238); 15(1.253) 508 047 
Level3 11(1.156); 15(2.426) 1.791 
Levelt 13(-2.02); 18(-1.760) -.926 
System behaviour Level2 13(-.793); 18(-.594); 110(.313); 111 (-.232) 070 -.014 
Level3 13(3.092) 3.092 
Levelt 12(-2.141); 16(-2.294); I7(-1.791); 112(-.512) -1.685 
System application Level2 12(1.112); 16.402); 17(.359); 112(.633) 627 -.009 
Level3 12(3.099); 16(2.263); 17(2.554); 112(1.519) 2.359 
Levelt 14(-1.656); 19(-.592); 113(-.828) -.575 
System evaluation Level2 14(.794); 19(1.034); 113(.888) 1.619 -.004 
Level3 19(2.629); 113(2.525) 2.577 


Difficulties Experienced with Regard to Different ST Skills for Ecological Issues; 
Ability to Identify Components and System Interactions 


The analyses of the students’ identification of components in the pond ecosystem (Item 1) indicated that 
most students (57.5%) correctly identified a moderate (Level 2) or high (Level 3) number of elements (Figure 3). 
However, one-fifth of the students could not correctly identify the system components. The analyses of students’ 
wrong answers indicated that they had some misconceptions regarding hidden elements; for example, the students 
confused ‘green algae’ with ‘bacteria’ and sorted ‘bacteria’ into the producer and ‘green algae’ into the decomposer 
categories. Furthermore, some students misclassified ‘bacteria’ as an abiotic factor in the pond ecosystem. 


Figure 3 
Distribution of Students’ Responses on Items 1 and 5 
Item 1 Item 5 
Level 3, Level 3, 
as 20.5 . soo 
2 Level 2, 37g Level 2, ae 
2 n=4-5 ~ 3<SX<.6 ; 
o n 
2 Level 1, 5 Level 1, 
2 13 = RF 1<SX< 3 = 
8 x 
~ Level 0, Level 0, 
13 20.3 SX< 1 40.3 
0% 10% 20% 30% 40% 0% 10% 20% 30% 40% 50% 
Proportion of students Proportion of students 


Note. n = number of components. The structural index (SX) = sum of all arrow strings, branches, and cycles / the number of 
concepts (Mehren et al., 2015). 


871 


https://doi.org/10.33225/jbse/23.22.865 LTT 


Journal of Baltic Science Education, Vol. 22, No. 5, 2023 


EXPLORING LOWER-SECONDARY SCHOOL STUDENTS’ SYSTEMS THINKING PERFORMANCE ISSN 1648-3898 /Print/ 
IN ECOLOGICAL ISSUES 
(PP. 865-880) ISSN 2538-7138 /oniines 


Although the students performed well in identifying the system components, analysis of the CMs indicated 
that most students possessed an incomplete picture of climate warming. Students’ abilities to identify interrelations 
among components were assessed using the structural index (SX) of the CM (see Figure 3). The results of the CMs 
showed that most students’ (82%) SX scores in Item 5 were less than .3 (Figure 3), indicating students’ low ability to 
identify interrelations among system components. Figure 4 shows three typical networks that visualise the struc- 
ture of the students’ CMs at the three levels. The first network in Figure 4 indicates that students at Level 1 could 
not make connections among all concepts provided in the CM and tended to identify direct linear connections 
among concepts; 14.2% of students reached Level 2, and students at this level could make indirect relations among 
concepts (e.g. branches [at least two arrows lead to/away from a concept]) and identify a simple carbon cycle via 
plant photosynthesis and respiration of organisms in the second network of Figure 4. Only a few students (3.5%) 
connected all the concepts in a networked manner. Complex relationships (e.g., different cycles and interactions 
among oxygen, plants, and carbon dioxide) can be identified at Level 3 (Figures 3 and 4). 


Figure 4 
Examples of the Networks in Students’ Concept Maps at the Three Levels 
Level1 students’ concept map Level2 students’ concept map 
Carbon Dioxide Industrial production 
Plants Mineral fules 


Animals & Human 


Oxygen Carbon Dioxide 


Pants 
Animals & Human Oxygén 


Level3 students’ concept map 


Industrial preduction 
Oxygen 


Animals & Human 
Carbor Dioxide 


Plants 


Mineral fules 
Decomposer 


Ability to Analyse System Behaviour and System Characteristics 


The results of Item 3 suggest that most students (54.1%) recognised dynamic changes among the three 
populations (wolves, deer, and plants), which help to maintain the equilibrium of the ecosystem. Furthermore, 
students’ responses to Item 8 also support the result that most students (64.6%) were able to explain the disap- 
pearance of grey wolves, which caused an increase in the number of deer, which, in turn, damaged plants. However, 
students’ inability to use emergent properties (e.g., carrying capacity and energy pyramid) to explain the changes 
in the ecosystem suggests that incomprehension of system characteristics hindered students’ analysis of dynamic 
behaviours in the system. Item 3 showed that only a few students (4.4%) could explain the long-term changes in 
the number of animals based on the conception of the energy pyramid (Table 4). 

Furthermore, the t-test analysis of Items 10 (M = .790) and 11 (M = 1.170) revealed that students performed 
significantly better in recognising the feedback loop than the nutrition cycle in the forest (¢ (2,147.7) = -8.92, p < 
.001), and only 39% of students could correctly connect the process of the nutrition cycle in Item 10. These find- 
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ings indicate that a lack of cyclic thinking makes it difficult for students to combine a variety of relationships into 
coherent cyclic and dynamic processes. Analysis of the CM also revealed a similar result; the third network in Figure 
4 suggests that different carbon cycles exist in the context of climate warming (e.g., interactions via photosynthesis 
and the respiration of plants, cyclical processes such as plants’ photosynthesis and the respiration of organisms, and 
the connection of mineral fuels to industrial emissions or human consumption). However, students at Levels 1 and 
2 (56.2%) only formed a fragmented perception of the carbon cycle in the context of climate warming (Figure 4). 


Table 4 
Item 3 System Characteristics and Contents 


System characteristics Specific contents Number of students 
Direct/indirect interactions The relations in food chains 254 (22.9%) 
Dynamics and stability Dynamic changes among populations 599 (54.1%) 
Emergent property Energy pyramid 49 (4.4%) 


Ability to Make Predictions and Decisions 


Figure 5 displays the proportion of student responses at different ST levels. The size of each chunk in the 
bar represents the proportion of the students. Figure 5 suggests that a large proportion of students’ responses 
were at Level 1, which implies that monocausal and direct thinking dominated the students’ systems application. 
Furthermore, students’ preference to identify direct effects closely related to the environment also supports this 
result; see, for instance, the common responses in Item 2: ‘Pollution of air and water caused the deaths of the 
organisms in the water’ Only a small proportion of students who formed a complex analysis at Level 3 (Figure 5) 
could anticipate the long-term effects based on a comprehensive understanding of the ecological, economic, and 
sociocultural dimensions of ecological issues (e.g., effects of industrial pollutants on residents’ income in Item 2 
and decision-making considering the sustainable development of the rainforest as well as the protection of local 
people's lifestyle in Item 12). 


Figure 5 
Students’ Response Distribution Regarding the Items in System Application and System Evaluation 
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System Application System Evaluation 


Note. Level 0 represents incorrect responses; Level 1 represents monocausal relations; Level 2 represents indirect/linear relations, 
and Level 3 represents complex relations. 
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Ability to Make Evaluations about the Structures and Applications of Systems 


The results of Item 4 showed that holistic thinking was lacking in students’ evaluation in that most students 
(84.3%) could find the error of isolated relations in the system, but few students (31.1%) could evaluate the relations 
from a system perspective. To make the right judgement about relation 9 (‘human consumption of crop’ is good 
for‘human health’) in Item 4, students had to think about the whole system instead of just the parts, because the 
overuse of pesticides would result in crops accumulating pesticide residues. However, more than half of the stu- 
dents (53.2%) overlooked this whole system background. Furthermore, the results for Items 9 and 13 suggest that 
most students made evaluations in a monocausal (Level 1)/incorrect (Level 0) manner (Figure 5). Further analysis 
of students’ responses indicated that these students lacked reflective consciousness about the limitations of pre- 
dictions and decisions and could not make full evaluations of the validity of decisions and predictions in a time 
dimension; for example, they were unable to determine the long-term effects of killing grey wolves on ecosystems 
and that of a fishing ban on the economy and society. 


Identification of Performance Gaps in Different Student Groups 


The t-test was used to compare the means of student performances using WLEs for particular student 
groups (Figure 6). Students showed significantly different levels of performance based on their school location 
(t (1,090) = -8.388, p < .001); rural students (M = -.236) showed lower performance than urban students (M = .187). 
The results suggested that there was no significant difference between male (M = .033) and female students 
(M = -.049; t (1,090) = 1.583, p = .114), and no significant difference between Grades 7 (M = .01) and 8 (M = -.02) 
students (t (1,090) = .672, p = .502). 

In the DIF analysis, Items 2, 5, 6, 7, and 9 appeared to be more difficult for rural school students, at a .01 
significance level. The specific content and ST skills regarding these items were examined, and the analysis of stu- 
dents’ responses indicated that rural students had more difficulties in identifying interactions among components 
and making predictions, decisions, and evaluations in an indirect or complex way than their urban counterparts. 
Furthermore, Items 5, 6, and 7 were all included under the context of climate warming, which means that this 
ecological issue is challenging for rural school students to understand. 


Figure 6 
Differences in ST Performance between Various Student Subgroups 
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Note. *** <.001. 
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Discussion 


This study incorporates a broader set of ST skills (e.g., system decision-making and evaluation) for assessing stu- 
dents’ consideration of diverse dimensions of ecological issues, resulting in a more comprehensive understanding of 
their ST performance in ecological issues. Furthermore, this study recruited a larger sample group than previous studies, 
thereby yielding more reliable and representative baseline data on lower-secondary school students’ performance. 

Regarding RQ1, the Wright map and the comparison of students’ abilities both suggest that, on average, the 
students reached Level 1 of ST about ecological issues. The low ST performance of the participants is not surprising 
because some previous studies from different contexts also found similar results (Ben-Zvi Assaraf & Orion, 2005; 
Mambrey et al., 2020). Among the four skills in ST, systems organisation was found to be relatively easier for students, 
which is consistent with the findings of Evagorou et al. (2009) and Hmelo-Silver and Pfeffer (2004), who observed 
that the structure of a system was the most cognitively easy for students to understand. 

The analysis of the students’ responses to the items in RQ2 further explained the reasons for their low perfor- 
mance in understanding ecological issues. Students exhibited difficulties comprehending the interrelations among 
system components and identifying hidden elements in the ecosystem. The microorganism is invisible and thus easily 
neglected, making it difficult for students to identify them (Ben-Zvi Assaraf & Orion, 2005). Results in this study align 
with previous research findings that emergence is the most difficult system characteristic for students to understand 
(Hmelo-Silver et al., 2007; Jin et al., 2019; Sommer & Luicken, 2010). However, the analysis implies that stability and 
dynamics are relatively easy for the lower-secondary school students in China. The ecosystem equilibrium highlighted 
in the Chinese lower-secondary school biology curriculum likely promotes students’ understanding of the character- 
istics. The little to no use of system concepts in explanations also supports the necessity of imparting general system 
characteristics (e.g., feedback loop, emergent properties, equilibrium) to scaffold students’ coherent understanding 
of complex systems from a system perspective (Gilissen et al., 2020; Gilissen et al., 2021). Students lacked the cyclic 
perception, as reported by Ben-Zvi Assaraf et al. (2005) for the water cycle, and most students were unable to draw 
a complete network of interactions. The lack of time dimension hinders students’ ability to make predictions and 
decisions based on long-term effects. 

Although evaluation is an essential aspect of ST (Fanta et al., 2020; National Research Council, 2010), it is usually 
ignored in assessment. This study's findings indicate that students were less proficient in system decision-making 
and evaluation. There may be several reasons for this. First, the absence of reflective consciousness made it difficult 
for students to contemplate the limitation of prediction and decision. Second, the lack of holistic thinking prevented 
most of them from deciding or evaluating from a comprehensive perspective. The results also suggest students’ in- 
ability to recognise other dimensions (such as social, economic, and cultural), and the indirect effects on ecological 
problems in their predictions, decision-making and evaluation. This further explains the dominance of monocausal 
reasoning in most students’ ST, and their inability to fully understand the causes of ecological problems. This led them 
to decide and evaluate from a unidimensional perspective. 

Regarding RQ3, this study's results contradict those of Jin et al. (2019), who concluded that the average per- 
formance of urban students was statistically significantly lower than that of rural students. A major reason for this is 
the contrast in urban area educational resources between China and the United States. Urban schools in the United 
States tend to have low resources, high teacher turnover, and low-quality students (Jin et al., 2019). However, with 
the accelerating urbanisation in China, resources have accumulated in urban areas. In general, urban schools in China 
have better educational resources and more highly qualified teachers than rural schools (Wen & Gu, 2017). Given that 
context-specific knowledge is necessary for developing ST (Ben-Zvi Assaraf & Orion, 2005; Sommer & Lucken, 2010), 
the lower performance of rural students in the context of global warming implies a possible lack of knowledge about 
this ecological issue. To explain the differences between rural and urban education more precisely, future research 
should explore the extent to which possible variables (e.g., socioeconomic status, teachers’ teaching quality, and 
students’ knowledge about ecological issues) influence students’ ST. 


Conclusions and Implications 
This study explored lower-secondary school students’ system thinking performance in ecological issues. The 
STTEI was administered for acquiring extensive data from Chinese students. Their overall ST performance was at 


Level 1, and an analysis of their responses revealed that they faced substantial difficulties in understanding ecologi- 
cal issues. Furthermore, this study is unique in revealing that most lower-secondary school students lack reflective 
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consciousness, and are thus unable to consider the indirect impact of diverse dimensions, leading to monocausal 
reasoning in system application and evaluation. This study also found that Chinese rural students did not perform 
as well as their urban counterparts, specifically in the context of climate warming, which may be explained by rural 
school students’ lack of the corresponding content knowledge. 

These findings have significant implications, particularly for developing lower-secondary students’ ST in 
ecological issues. First, teachers must address students’ shortcomings, especially regarding emergent properties, 
cyclic perception, and interactions within complex systems. Second, ecological issues must be discussed from 
multiple perspectives for promoting ST in science classes and improving students’ causal reasoning in complex 
systems. Third, there should be greater efforts in making rural students understand climate change. Future research 
must examine other variables for developing teaching strategies suited to students’ backgrounds. Furthermore, 
qualitative methods (e.g., semi-interview) should be used for future assessments, so that the underlying causes of 
students’ low performance are identified. 
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Sunlight 


Item 1 Answer the following questions based on the information 
provided in the picture: 


Organisms that use the energy of sunlight to produce organic matter 
include 


Abiotic factors affecting the pond ecosystem include 


@ am 
SSS 


os 
Large Fish 


small He 


The decomposer in the pond is 


Magnified View 
of Green Algae 


seemed ics (Modified from the National Assessment of Educational Progress, 
cuore http://nces.ed.gov/NATIONSREPORTCARD/) 


Item 2 A newly built factory near the pond emits substantial amounts of sulphur dioxide and other pollutants and 
discharges wastewater containing heavy metal pollutants, nitrogen, and phosphorus (nitrogen and phosphorus 
will cause many green algae to multiply) into the pond. What do you think this will do to the creatures in the pond 
and the surrounding villagers? 


Item 3 A remote island is uninhabited. The main animals on the island are wolves and deer, and the island is free 
from external interference all year round. 


(1) Select the answer below that best describes what will happen to the average populations of the animals over 
time. ¢ ) 
a) Thedeer will all die or be killed. 
b) The wolves will all die. 
c) There will be many more deer than wolves. 
d) There will be many more wolves than deer. 
e) The populations of each will be approximately equal. 
(2) Choose the reason for your answer ( ) 
a) Wolves prey on deer, and deer are eaten up by wolves. 
b) Wolves prey on deer, causing the number of deer to decrease. 
c) Wolves prey on deer. The number of deer decreases. Wolves die as they lack food. 
d) Tomaintain the ecological balance of the ‘plant > deer > wolf’food chain, the number of wolves and 
deer will fluctuate up and down within a certain range. 
e) When energy is transmitted along the food chain, it decreases step by step. The population of wolves 
store less energy than the population of deer. 


Item 4 The following concept map describes the impact of heavy use of pesticides in farmland on people and the 
ecosystem. There are two errors in the word relationships described in the concept map. Write the wrong serial 
number and explain the correct relationship. 


large use 
of 
pesticide 


10.result in 


l.result in 


water 
pollution meee le 
P 3 ~ [zeduction 3.result in 
— 


14! ss 2. promote 
harmful to the number of 
growth of natural enemies of 
crops pests decreases 
human \ v4 
health 5. increase 6. promote 4. in favour 
4 of 
crop 
yield BS breed of 
9.in favour or 8. promote 


of 


7. increase 
economic 
human growth 
consumption 


TUS https://doi.org/10.33225/jbse/23.22.865 


Journal of Baltic Science Education, Vol. 22, No. 5, 2023 


ISSN 1648-3898 /Print/ EXPLORING LOWER-SECONDARY SCHOOL STUDENTS’ SYSTEMS THINKING PERFORMANCE 
IN ECOLOGICAL ISSUES 
ISSN 2538-7138 /oniiney (PP. 865-880) 


Item 6 Together with the concept map you have constructed, please predict what will happen if the number of 
plants substantially decreases. 


Item 7 Everyone is responsible for protecting the Earth as our home. Please put forward specific measures to miti- 
gate climate warming. 


Please answer the following questions according to the information in the food web (the arrow in the food web 
points to the predator). 


Item 8 lf all grey wolves disappear, what will be the trend in the 
deer population ( ) 


grey 
wolves 


eagles 


A. It will keep growing. C. It will keep decreasing. 


B. It will grow first, then decrease. _ D. It will decrease first, then 
grow again. 


deer 


Explain the reasons for your answer: 


insects 


polar 


Item 9To protect the safety of villagers near the forest, some people proposed to kill all the grey wolves in the for- 
est, but Ming argued that ‘if there are no grey wolves in the forest, the number of all herbivores in it will increase 
rapidly in a short period, thus damaging the stability of the ecosystem’ Do you think Ming's analysis is correct? 
Please explain your reasons. 


Item 10 Use arrows to connect the following words to form the nutrient cycle of poplar trees in the forest. The 
first arrow has been provided in the picture, and the last arrow you draw must point to ‘poplar trees’ 


Falling of leaves 


Absorption of nutrients 


via the roots 


Release of nutrients 


Decomposition by fungi 


and bacteria in the soil 


Plant growth from 
nutrients 


Poplar trees 


(Modified from Mehren et al., 2018) 
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Item 11 Trees and plants can reduce ambient temperature through transpiration. The higher the ambient 
temperature, the stronger the transpiration of plants, and the more water in plants is emitted into the air. Please 
judge what changes will happen to the forest environment when the temperature rises and the transpiration of 
plants increases. Fill in ‘promote’ or‘weaken at A, B, and C in the following figure (as shown in the example below). 
Please fill in the blanks in the order of A> B > C. 


Item 12 The Khmus people in Xishuangbanna have been using slash-and-burn cultivation in the tropical 
rainforest. They cut down and burn trees to clear the land for planting food. The ashes from the burning of trees 
can increase soil fertility. This planting method is only effective for one year, and the land fertility will decline in the 
next year. The rainforest includes valuable trees (with high prices) and ordinary trees. After cutting them down, it 
takes at least ten years to recover the trees. 


(1) To achieve sustainable development of tropical rain forests and the planting activities of Khmus, which of the 
following measures do you think is the most reasonable? 


A. Cut down all the trees in the rainforest and sell the valuable trees. 
B. Only cut down the valuable trees in the rainforest and sell all the valuable trees. 


C. Divide the entire rainforest area into ten equal segments. Every year, cut down trees only in a partial 
area and sell the resulting valuable wood. 


D. Divide the entire rainforest area into ten equal segments. Every year, cut down trees only in a partial 
area and sell all the resulting wood. 


(2) Explain the reasons for your answer. 


Item 13 Zhoushan in Zhejiang Province is known as the ‘Eastern Fish Warehouse’ in China. People here basically 
rely on fishing to make a living. However, with the overfishing of yellow croakers, the number of yellow croakers 
has declined sharply in the past few years. One reason is that many young yellow croakers have been caught. To 
prevent the extinction of wild yellow croakers, the measure of ‘banning fishing within three years’ was proposed. 
Do you think this measure is feasible? Please explain your reasons. 
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GENDER DIFFERENCES IN 
EXPERIENCES OF 6TH-GRADE 
STUDENTS INQUIRY ACTIVITIES 
IN THE REPUBLIC OF KOREA 


Soo-min Lim, 
Youngshin Kim 


Introduction 


Individuals have various experiences, and depending on the way they 
construct and interpret the meaning of their experiences, they determine 
various future behavioral directions. The same goes for learning. In science 
education, it is the experience of learners directly participating in physical 
activities and thinking activities related to science and includes learners’emo- 
tions and subjective thoughts based on the experiences of classes or inquiry 
activities in the school curriculum (Choi & Choi, 2012). These experiences 
can play an important role in the formation of learners’ scientific knowledge. 

The most representative class in science class is inquiry activity. Inquiry 
activities are emphasized to encourage active participation and interaction 
among students in science classes (Nam et al., 2002). They are favored by 
students (Lim et al., 2021), but they are also a reason why students dislike 
science classes (Lee et al., 2007; Kim & Yang, 2005). In other words, the criti- 
cal factor that determines success or failure in science education is inquiry 
activities (Kang et al., 2007). Small-group inquiry activity is one method for 
successfully completing inquiry activities. In this activity, students plan the 
process from designing experiments to drawing conclusions by interacting 
with other members and experience the problem-solving process by pre- 
senting, accepting, criticizing, modifying, and discussing opinions (Lumpe 
& Staver, 1995; Richmond & Striley, 1996). 

Many studies have analyzed the effects (Gillies, 2008; Johnson & John- 
son, 2003; Slavin, 2013; Stevens, 2003), role types (Maloney, 2007; Richmond 
& Striley, 1996), and interactions among students in small-group inquiry 
activities (Alexopoulou & Driver, 1996; Kim et al., 2017; Lim et al., 2020). 
However, these studies failed to analyze what kind of activities and actions, 
thoughts, consciousness, and emotions were experienced by students in 
each stage of the scientific inquiry activities. 

Meanwhile, Problem-solving strategies or interactions in small-group 
inquiry activities vary depending on gender (Lim et al., 2020; Peltz, 1990), 
academic achievement, interest, and affective aspects (Alexopoulou & Driver, 
1996; Yang et al., 1996; Yang et al. al., 2006). In particular, it is reported that 
there is a very clear gender difference in attitudes related to science (Park 
& Shin, 2011). These differences can have a negative impact on successful 
scientific inquiry activities. Therefore, a study on the causes of gender dif- 
ferences in small group inquiry activities is required. 
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Abstract. Each student's individual 
experience in learning is very important. 
Lower secondary school students are 
engaged in activities that focus on 

inquiry rather than the acquisition of 
scientific knowledge in science classrooms. 
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Students experience various events, interactions, activities, and thoughts at every moment during small-group 
inquiry activities. If an analysis is conducted on the interaction or experiences of each stage in the small group 
inquiry activity, that is, the activities and thoughts, it is possible to find the cause of the gender difference as well 
as the specific implications for the success of the small group inquiry activity. Therefore, it is necessary to analyze 
what kind of activities and actions, what kind of thoughts, and what kinds of consciousness and emotions are 
experienced in each stage of scientific inquiry activities. These experiences are mainly divided into external and 
internal experiences (Hektner et al., 2007). External experience refers to events experienced externally (Hektner 
et al., 2007), whereas internal experience refers to events experienced internally (Kim et al., 2005). The experience 
sampling method (ESM) is used to determine external and internal experiences. 

ESM can analyze what kind of activities and emotions that participants feel under what circumstances at a 
specific time (Hektner et al., 2007). It has been used in various cultures for a long time and is recognized for its 
reliability and validity (Park & Choi, 2018). ESM can reduce memory bias because it immediately secures informa- 
tion at a specific moment and is suitable for analyzing various psychological changes that students experience. In 
particular, ESM needs to be applied when the participant is a student because it is difficult to maintain consistency 
in subjective judgment on cognition or emotion (Savin-Williams, 1987). 

ESM is actively applied in psychology or empirical research for professional development. In the field of edu- 
cation, a previous study examined the link between the quality of a cooperative learning environment and the 
quality of students’ experiences in subjects such as English, mathematics, science, and sociology (Shernoff et al., 
2016). Another study analyzed the experiences of leaders and alienated students in inquiry activities (Choi et al., 
2022). Therefore, it is useful to analyze the experience of general students according to gender. 

In lower secondary school, emphasis is placed on improving the inquiry process rather than scientific knowl- 
edge (Haefner & Zembal-Saul, 2004), and many inquiry activities centered on observation or simple experiments 
are performed. The small-group science inquiry activities experienced by lower secondary school students during 
this period determine very diverse perceptions, such as their future jobs related to science and technology or their 
dislike for science (Kim & Shin, 2013). Therefore, it is required to have an interest in science and a correct percep- 
tion of it in the lower secondary school age (Korean Association of Career Education, 2000), which is the stage in 
which students begin to recognize their career path. Therefore, it is meaningful to analyze the experiences that 
lower secondary school students experience in small-group science inquiry activities. 

In this study, ESM was used to compare external and internal experiences according to gender in small-group 
science inquiry activities in lower secondary schools. External experiences were divided into the main activities and 
actions, and internal experiences were divided into thoughts, contributions of actions, consciousness, and emo- 
tions, and these were compared and analyzed according to gender groups. To this end, the differences between 
male and female students were analyzed by comparing the external and internal experiences experienced by both 
groups at each stage of scientific inquiry activities (observation, variable control, and conclusion drawing). Through 
this study, it will be possible to provide guidance for inquiry activities by identifying the behavior and thoughts of 
lower secondary school students in the inquiry stage. 


Theoretical Background 
Experiences 


The daily life experienced by individuals, which consists of trivial things being repeated every day, has been 
taken for granted. However, from the beginning of emphasizing the specific reality and life of individuals in ev- 
eryday life, experience has become the subject of academic research (Han & Son, 2009). Individuals, who are the 
subject of experience, experience various events and activities, and through these external activities, individual 
subjective evaluations are made at each moment, resulting in various internal experiences. These experiences 
are settled and accumulated in various forms in human memory. Experience is not simply linear, but it takes on 
a complex character that is influenced by previous experiences and also affects future experiences. Beyond this 
temporal sequence, it is reconstructed and reorganized. 

Experience can be divided into external and internal experiences (Csikszentmihalyi, 1990). External experience 
means “an experience that one actually experiences or experiences externally.’ Internal experience can be defined 
as“an experience experienced internally in the mind or heart.’The former is related to “where, with whom, and what 
we did,’ whereas the latter refers to psychological factors related to “what we felt and thought” in such activities. 
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In the case of external experiences, previous studies usually classify into the type of activity (Lee & Choi, 2011), 
the people with whom they are together (Hnatiuk, 1991; Larson et. al, 1986; Lee & Choi, 2011), and the current 
place (Larson & Richards, 1994; Schneider & Waite, 2005). In the case of inquiry activities, since the current place 
is the same, the type of activity, and the people with whom they are together were used for classification. At this 
time, Kim et al. (2005), it was conducted by reflecting the characteristics of science inquiry activities. 

On the other hand, although we have various external experiences, what actually determines our consciousness 
and behavior is the internal experience of how our emotions react when we do these actions (Csikszentmihalyi, 
1990). Likewise, although external experiences are important, it can be said that an individual’s internal experi- 
ences in the process of life are very meaningful to us (Fredrickson, 2009). Accordingly, the internal experience 
was categorized into what kind of thoughts were made during inquiry activities, whether one thought that one's 
actions contributed to inquiry activities (Csikszentmihalyi & Schneider, 2000), and what kind of consciousness and 
emotions were felt during inquiry activities (Shernoff et al., 2016). 

Previous studies on experience classified quality of life into internal and external experiences (Kim & Kim, 2019; 
Lee, 2017) and analyzed students’ class environment (Park, 2018; Cho & Kim, 2018). In science education, studies 
have analyzed the influence of science awareness and experience on career choice in science and technology 
(Lee, 2011) and conducted qualitative research on self-efficacy according to science teachers’ experiences (Park, 
2001). In addition, some studies have surveyed the impact of informal science experiences outside the classroom 
(Bell et al., 2009; Galen, 1993; Parker & Gerber, 2002) and science experiences through mentorship with experts in 
the field of science (Feldman, 2007; Koch & Appleton, 2007; Markowicz, 2004) on students’ cognitive and affective 
aspects. However, these studies did not analyze students’ actual experiences during scientific inquiry activities. 


The Experience Sampling Methods 


A person's behavior is what appears in the interaction with the situation (Oishi et al., 2004). ESM was developed 
by Csikszentmihalyi and Larson as an effective way to properly identify changes in human behavior depending 
on the situation. Several previous studies (Brandstatter, 1983; Flory et al., 2000; Oishi et al., 2004) revealed the 
complexity of the interaction between situations and humans, all of which used ESM. Since ESM requires immedi- 
ate responses at random points in real life, it has the advantage of being able to analyze complex interactions of 
situations, behaviors, and emotions. 

It is a psychometric method in which participants’ actions are stopped at specific points and their experiences 
are recorded in real time. The ESM is classified into three types according to the response signal notification method: 
interval-contingent sampling, signal-contingent sampling, and event-contingent sampling (Reis, & Gable, 2000). 
Interval-contingent sampling is a method of responding at predetermined intervals. Signal-contingent sampling is 
a method of responding according to a randomly notified signal without fixing the timing. Although it can reduce 
the psychological distortion of the respondent, it has a problem of increasing the burden on the respondent. Event- 
contingent sampling is a method of responding when a predetermined event occurs. Event-contingent sampling 
is effective when analyzing events with a very low frequency of occurrence (Cho & Nam, 2005). Likewise, because 
participants’ experiences are sampled multiple times, a reasonable understanding of the phenomena experienced 
by individuals can be obtained, and changes in participants’ experiences can be observed over time (Stone et al., 
1999). It is also a research method with very high ecological validity that immediately grasps what an individual 
feels and thinks on their own in a state where the intervention of the observer is minimized without resorting to 
reminiscence (Csikszentmihalyi & Larson, 1987). Therefore, in previous studies on ESM, the internal and external 
experiences experienced by general students (Choi et al., 2003; Csikszentmihalyi & Graef, 1980; Kim et al., 2005; 
Shernoff et al., 2016), gifted children (Choi & Choi, 2012), and adults (Han & Son, 2009) regarding small sample of 
a specific layer were measured in specific situations, and the resulting quality of life or emotions were analyzed. In 
other words, ESM records and analyzes the experience or emotion of the period presented by the researcher ina 
questionnaire distributed in advance to the subject (Hektner et al., 2007). 

ESM compensates for the disadvantages of existing research methods, such as memory errors caused by 
recall, and allows the observer to directly record the behavior and emotions of subjects, preventing the involve- 
ment of the observer's subjectivity. In addition, as a method to effectively identify changes in human behavior 
according to various situations, ESM can analyze the complex interaction between situations, individual behavior, 
and emotions (Shin, 2010). 

Experiential knowledge helps solve problems in new and uncertain situations by providing information about 
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possible limitations in solving the problem, what to focus on, or what not to do (Kolodner, 1997). In this way, stu- 
dents’ experiences can provide implications for teaching-learning strategies. In particular, by analyzing students’ 
experiences in science inquiry activities, which are the most representative form of class in science subjects, it is 
possible to provide implications for the success of science inquiry activities and science learning. Therefore, it is 
very useful to measure the behaviors and emotions that students typically experience during scientific inquiry 
activities by ESM. 


Research Methodology 
Design 


This study was conducted using a quantitative research design to answer the research question. The quan- 
titative research design involves collecting quantitative statistical data on several variables studied to answer the 
research questions (Cohen & Manion, 1994; Creswell, 2003). This study is a quantitative research design that collects 
data using questionnaires. 

The independent variable in this study was gender, and the dependent variables were external and internal 
experiences during inquiry activities. Quantitative data of this study were collected by modifying and supplement- 
ing the questionnaire developed by Kim et al. (2005) on external experience and the questionnaire developed by 
Csikszentmihalyi and Schneider (2000) and Shernoff et al. (2016) on internal experience. 

The study was conducted from October to December 2018. The participants were students from three schools 
located in metropolitan cities and municipalities in the Republic of Korea and were sixth-grade lower secondary 
school students who were not gifted or special students. 

Under the scope of this study, the participants were invited through the homeroom teacher at the lower 
secondary school. All of the participants and their parents agreed to participate voluntarily in the research. While 
collecting the data, the participants did not write their names and last names while responding to the items. 


Participants 


The participants of this study were 6th-grade students. Students from two schools located in a metropolitan 
city with a population of 2.5 million and one school located in a city with a population of 300,000 in the Republic of 
Korea were sampled. A total of 77 students (40 male and 37 female students) were sampled. Students who agreed 
on the purpose and method of the study were targeted, and the science achievement level of these students was 
moderate. By the central limit theorem, this study was conducted on the basis that quantitative verification is 
possible because if the sample for quantitative verification is large enough (n>30), independent, and sampled at 
random, it exhibits a normal distribution regardless of the distribution (Kwak & Kim, 2017). 


Inquiry Programs 


The program used in this study is a candle-burning experiment. In this inquiry activity, a candle is set in clay 
in a water tank, and a glass cup is placed over the candle. Students observe the rising water level in the glass as the 
candle goes out. Through observation, an inquiry problem is given, and an experiment is designed and conducted 
to solve the given inquiry problem. The reason for selecting this topic is that it is not covered in lower secondary 
schools in the Republic of Korea, so it is possible to conduct research activities without prior knowledge. The total 
time of the program was 100 min (40 min for the observation stage, 25 min for the variable control stage, 25 min 
for the conducting experiment stage, and 10 min for the conclusion drawing stage). 

The stages of inquiry activity consist of observation, variable control, and conclusion drawing. In the observa- 
tion stage, students directly observed the candle burning and described various observations. After the observation 
activity, the classroom teacher presented the research topic “the amount of water rising in the collector accord- 
ing to the number of candles.’ In the variable control stage, the experimental design was designed to verify the 
research topic presented through interaction among the group members. At this stage, the group members were 
asked to discuss with each other how to control the variables and carry out the experiment in detail. According to 
the experimental plan, each small group conducted the experiment, collected the data, converted the data, and 
drew conclusions based on the experimental results. 
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Questionnaire 


ESM was used to analyze external and internal experiences in scientific inquiry activities. The questionnaire 
surveys students’ external and internal experiences in three stages: observation, variable control, and conclusion 
drawing. 

The questionnaire consisted of the main activities and main actions as the external experience and the main 
thought and contribution of actions during the inquiry activities as the internal experience. For internal experi- 
ence, questions about consciousness and emotions felt during inquiry activities were included. Table 1 presents 
the detailed composition of the questionnaire items. 


Table 1 
Composition of the Questionnaire 


Experience Item numbers Contents Item type 
1 Main activities Closed-ended 
External experiences 
2 Main actions Open-ended 
3 Main thoughts Open-ended 
4 Contribution of actions Open-ended 
Internal experiences 
5 Consciousness* Likert scale 
6 Emotions* Likert scale 


*survey after the scientific activity 


The items of the main activities and actions of external experience were based on the study by Kim et al. 
(2005) with some modifications. The main activities in the external experience were structured in closed-ended 
questions so that students could present and choose from listening to the teacher's words or directives, conduct- 
ing the experiment with their peers, talking about the experiment, and personal behavior (Kim et al., 2005). The 
main actions were configured in open-ended questions so that the participant could record what kind of action 
and with whom it was conducted during the inquiry activity. 

The main thoughts and contributions of the main actions of internal experience were constructed based on 
the form used by Csikszentmihalyi and Schneider (2000). Consciousness and emotion during inquiry activities were 
used by modifying the questionnaire by Shernoff et al. (2016). The main thought in the internal experience was 
made possible to present all of the thoughts that were in each stage of the inquiry activity and to present activities 
related and unrelated to the class. The contribution of actions made it possible to suggest whether the actions of 
the participant were helpful to the inquiry activity (Shernoff et al., 2016). In addition, in the internal experience, 
the consciousness and emotion felt during the inquiry activity were presented on a five-point scale. 

Six experts were consulted to verify the validity of the test items. Among them, four had doctoral degrees in 
science education, and two were science teachers with master’s degrees and more than 10 years of experience. The 
content validity of the item was 90.4%. The Cronbach's alpha and reliability of the questionnaire for consciousness 
and emotion were .939 and .872, respectively. 


Analysis Framework 


To analyze the open-ended items in the questionnaire, students’ external and internal experiences were 
categorized into experimental activities (A), emotions (B), interactions (C), and others (D) sections based on the 
questionnaire’s contents (Table 2). The experimental activities section (A) selected observation, measurement, 
variable control, result prediction, data transformation, and conclusion drawing in the basic inquiry and integrated 
inquiry among the inquiry process elements of the Test of Science Process Skills. Experiment preparation (A1) and 
conducting experiment (A2) were added because the activities of preparing and conducting experiments through 
interaction among students within a small group are important activities. 

The emotion section (B) was classified into positive and negative emotions. Positive emotions are fun, enjoy- 
able, exciting, interesting, surprising, and want to do more experiments, whereas negative emotions are futile, 
annoying, difficult, boring, and uninteresting. 
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The interaction section (C) was classified into questions, responses, presentation of opinions, receiving opin- 
ions, behavioral participation, atmosphere, and responses to opinions (Lee et al, 2002). Experiences that were not 
included in the above classification and actions, thoughts, and non-responses that were not specifically related to 
inquiry activities were classified in the others section (D). The validity of the analysis framework was obtained by 
consulting five science education researchers, and a validity of 88% was obtained. 


Table 2 
Framework for Analysis 


Experiment preparation(A1), conducting experiment (A2), observation(A3), measurement(A4), variable control(A5), 


Expentnienial aeuvilies (A) result prediction(A6), data transformation(A/), data interpretation(A8), conclusion drawing(A9) 


Emotions (B) Positive emotion(B1), negative emotion(B2) 


Question(C1), response(C2), presentation of opinions(C3), receiving opinions(C4), behavioral participation(C5), 


interactions KC} atmosphere(C6), responses to opinions(C7) 


Non-response, actions, and thoughts not related to inquiry activities (e.g., personal action, small talk, thinking about 


Chee) games, thinking about break time, etc.) 


Data Collection and Analysis 


People’s experiences emerge through their interactions with situations (Oishi et al., 2004). In other words, 
students’ experiences emerge in the interaction with the situation of inquiry activities. In order to collect vari- 
ous internal and external experiences experienced by students, the small group was randomly composed of five 
members. According to Lim et al. (2019), the most diverse forms of interaction appear in a small group consisting 
of five members, it was assumed that various internal and external experiences could be collected. According 
to the science block time, the science inquiry activities took three classes, each lasting 40 minutes. At this time, 
event-contingent sampling, which is the most efficient way to sample students’ experiences in the special situation 
of scientific inquiry, was used (Cho & Nam, 2005). In the Republic of Korea, most participants in lower secondary 
schools, including science, are taught by homeroom teachers. Lower secondary school education in the Republic 
of Korea is centered on the class, and the homeroom teacher is responsible for managing each class. Homeroom 
teachers in lower secondary schools, guide students in subject learning, help them adjust to society, provide evalu- 
ation and feedback, and provide guidance and counseling on life attitudes. This inquiry activity program was also 
conducted by the homeroom teacher. Before the inquiry activity program was conducted, the homeroom teacher 
held a seminar and practiced on the program treatment process in advance. 

When each step of the inquiry activity was completed, an alarm was sounded, and each step-by-step ques- 
tionnaire was answered for 10 min. Among the questionnaires collected from the students, closed-ended items 
were analyzed based on frequency analysis, and open-ended items (item 2 to item 5) were analyzed based on the 
analysis framework (Table 2). Four doctors of science education used the analyzed data to determine agreement, 
and 92% agreement was obtained among them. 

The collected data were quantitatively analyzed using SPSS 25.0. The open-ended items were analyzed ina 
classification frame, followed by descriptive statistics, and the x” test was used to compare male and female students 
in each stage of the inquiry activity. In the case of the five-point Likert scale items, differences were compared and 
analyzed through descriptive statistics and independent sample t-test. In the case of negative items, analysis was 
performed through reverse scoring. 
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Research Results 
External Experience 
Main Activities 
The external experience in the inquiry activity stage was largely classified into main activities and main ac- 
tions. Table 3 shows the main activities that students performed in the stages of observation, variable control, and 


conclusion drawing among scientific inquiry activities. This allows for multiple responses. 


Table 3 
External Experience regarding the Main Activity at Each Stage of Inquiry 


Observation: n(%) Variable control: n(%) Conclusion drawing: n(%) 
Male Female Total Male Female Total Male Female Total 
Listening to the teacher’s 12 14 26 12 12 24 11 10 21 
directives (21.4) (22.2) (21.8) (21.1) (22.2) (21.6) (18.0) (17.2) (17.6) 
Experiment with peers 29 30 59 21 14 35 29 25 54 
(51.8) (47.6) (49.6) (36.8) (25.9) (31.5) (47.5) (43.1) (45.4) 
Conversation with peers 12 17 29 18 20 38 16 17 33 
about experiments (21.4) (27.0) (24.4) (31.6) (37.0) (34.2) (26.2) (29.3) (27.7) 
Personal action 3 0 3 4 2 6 3 3 6 
(5.4) (0.0) (2.5) (7.0) (3.7) (5.4) (4.9) (5.2) (5.0) 
Others 0 2 2 2 6 8 2 3 5 
(0.0) (3.2) (1.7) (3.5) (11.1) (7.2) (3.3) (5.2) (4.2) 
Total 56 63 119 57 54 111 61 58 119 


(100.0) (100.0) (100.0) (100.0) (100.0) (100.0) (100.0) (100.0) +— (100.0) 


Experiments with peers (49.6%) was the main activity performed by students in the observation stage, fol- 
lowed by conversation with peers about experiments (24.4%) and listening to the teacher’s directives (21.8%). 
Conversation with peers about experiments (34.2%), experiments with peers (21.4%), and listening to the teacher's 
directives (21.6%) were the main activities in the variable control stage. Meanwhile, experiments with peers (45.4%), 
conversation with peers about experiments (27.7%), and listening to the teacher's directives (21.6%) were the main 
activities in the conclusion drawing stage. 

The following gender differences were observed in each stage of the inquiry activity. For male students, experi- 
ments with peers was the main activity in the observation stage, followed by listening to the teacher's directives 
and conversation with peers about experiments (both having the same frequency). For female students, experi- 
ments with peers was the main activity, followed by conversation with peers about experiments and listening to 
the teacher's directives. As such, experiments with peers showed the highest frequency regardless of gender in 
the observation stage. In the variable control stage, experiments with peers was the main activity for male stu- 
dents, followed by conversation with peers about experiments and listening to the teacher's directives. For female 
students, conversation with peers about experiments was the main activity, followed by experiments with peers 
and listening to the teacher's directives. In the conclusion drawing stage, experiments with peers was the main 
activity for male and female students, followed by conversation with peers about experiments and listening to 
the teacher’s directives. 

The results showed that experiments with peers was the main action performed in most inquiry activity stages. 
However, conversation with peers about experiments had a higher frequency for both genders in the variable 
control stage compared with other stages. During the inquiry activities, students mostly conducted experiments 
with peers, conversed with peers about experiments, and listened to the teacher's directives. However, the fre- 
quency of conversation with peers about experiments was relatively higher in the variable control stage compared 
with other stages. This is expected to be more active among students in the variable control stage, in relation to a 
previous study (Choi et al., 2022) that reported that verbal interactions occur more actively in the variable control 
stage than in other stages. 
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Main Actions 


The main actions performed by lower secondary school students in each stage of the science inquiry activities 
are as follows (Table 4). In the experimental activity section (A) during the observation stage, the students’ main 
activities were conducting experiments or observing. In the emotion section (B), only positive emotions appeared. 
In the interaction section (C), behavioral participation (C5), presentation of opinions (C3), and behavior related to 
the atmosphere (C6) were in that order. 

In the experimental activity section (A) during the variable control stage, students usually prepared experi- 
ments or control variables as their main actions. In the emotion section (B), only negative emotion (B2) appeared 
unlike that in the observation stage. In the interaction section (C), the presentation of opinions (C3) and receiving 
opinions (C4) had the highest frequency. In the variable control stage, the frequency of behavior related to inter- 
action was higher than that in other stages. In particular, there were many practical verbal interactions in which 
opinions were exchanged on inquiry activities. 

In the conclusion drawing stage, most actions were related to the experimental activity section (A), followed 
by actions related to the interaction section (C) and actions related to the emotion section (B). 


Table 4 
Main Actions of External Experiences at Each Stage of Inquiry 


Observation: n(%) Variable control: n(%) Conclusion drawing: n(%) 


Sections 
Male Female al Male Female Val Male Female al 
Al 4(1.1) 1(0.9) 19(21.6) — 15(14.4) 7(9.3) 4(4.6) 
A2 —31(35.6) —-30(28.3) 3(3.4) 5(4.8) 29(38.7)  28(32.2) 
AZ 14(16.1) 15 (14.2) 0(0.0) 0(0.0) 8(10.7) 9(10.3) 
A4 2(2.3) 1(0.9) 0(0.0) 0(0.0) 7(9.3) 6(6.9) 
Experimental AS 0(0.0) 0(0.0) 13(14.8)  11(10.6) 4(1.3) 4(1.1) 
activities AG 0(0.0) 3(2.8) 3.345 5(5.7) 6(5.8) 1.113 4(1.3) 0(0.0) 3.650 
) AT 0(0.0) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 4(1.1) 
A8 0(0.0) 0(0.0) 0(0.0) 0(0.0) 4(1.3) 4(1.1) 
Ag 0(0.0) 0(0.0) 0(0.0) 0(0.0) 3(4.0) 4(1.1) 
ve 48(55.2)  50(47.2) 40(45.5)  37(35.6) 57(76.0) 51(58.6) 
Bt 2(2.3 3(2.8 0(0.0 0(0.0 4(1.3) 2(2.3) 
Emotions B2 0(0.0) 0(0.0) on 0(0.0) 3(2.9) af 0(0.0) 4(1.1) aa 
ed 2(2.3) 3(2.8) 0(0.0) 3(2.9) 4(1.3) 3(3.4) 
Ct 0(0.0) 2(1.9) 0(0.0) 2(1.9) 0(0.0) 0(0.0) 
C2 0(0.0) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 
C3 4(4.6) 13(12.3) 16(18.2)  22(21.2) 2(2.7) 7(8.0) 
vitae 2 C4 2(2.3) 7((6.6) 17(19.3)  21(20.2) 2(2.7) 5(5.7) 
(C) C5 —«-.20(23.0)~—S—«18(17.0) 7.486 4(4.5) 7(6.7) 4.655 6(8.0) 13(14.9) 1.415 
C6 7(8.0) 7(6.6) 2(2.3) 7(6.7) 3(4.0) 5(5.7) 
C7 3(3.4) 5(4.7) 4(4.5) 2(1.9) 3(4.0) 3(3.4) 
a 36(41.4)  52(49.1) 43(48.9)  61(58.7) 16(21.3) 33(37.9) 
Others (D) 4(1.1) 1(0.9) ie 5(5.7) 3(2.9) 4(1.3) 0(0.0) 
a 87 106 88 104 75 87 
(100.0) (100.0) 5 (100.0) (100.0) 7 (100.0) (100.0) a 
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In the observation stage, actions related to experimental activities had the highest frequency for male students, 
followed by actions related to interactions and actions related to emotions. On the other hand, actions related to 
interaction had the highest frequency for female students, followed by actions related to experimental activities 
and actions related to emotion. In the observation stage, actions related to experimental activities and interactions 
had the highest frequency regardless of gender. Specifically, male and female students conducted experiments 
and observations in relation to experimental activities. In the interaction section, actions related to behavioral 
participation had a high frequency regardless of gender. There was no difference in the main actions of external 
experiences at the observation stage of inquiry by gender (p>.05): In the experimental activities section (y? = 3.345, 
df= 8, p=.911), interactions section (y? = 7.486, df = 6, p = .278), and emotions section (y? = .000, df = 1, p =1.000). 

In the variable control stage, actions related to interaction had the highest frequency regardless of gender, 
followed by the actions related to the experimental activities section (A). Specifically, actions related to presenting 
opinions and receiving opinions had a high frequency regardless of gender. The actions related to the experimental 
activities section, experiment preparation and variable control activities were mainly performed. In this way, there 
was no difference in the main actions of external experiences at the variable control stage of inquiry by gender 
(p>.05): In the experimental activities section (y? = 1.113, df = 8, p = .997), interactions section (y? = 4.655, df = 6, 
p =.589), and emotions section (? = .000, df= 1, p =1.000). 

In the conclusion drawing stage, actions related to the experimental activities section (A) had the highest 
frequency regardless of gender, followed by actions related to interaction and emotion section (B). Moreover, 
actions related to the experimental activities section (A) had a higher frequency in the conclusion drawing stage 
compared with other stages, confirming that actions related to interactions had a low frequency. On the other 
hand, it was confirmed that students did not perform activities related to conclusion drawing, such as conducting 
experiments or conducting observations, in the conclusion drawing stage. In this way, there was no difference 
in the main actions of external experiences at the conclusion drawing stage of inquiry by gender (p>.05): In the 
experimental activities section (7? = 3.650, df = 8, p = .887), interactions section (y? = 1.415, df = 6, p = .965), and 
emotions section (y? = .444, df= 1, p =.505). 

In the observation and conclusion drawing stages, actions related to the experimental activities section (A) 
had the highest frequency. However, in the variable control stage, actions related to interactions had the highest 
frequency. In the variable control stage, verbal interaction was found to facilitate the exchange of opinions. This 
result proves that the variable control stage does not need to be focused on increasing student interaction. However, 
negative emotions were most often felt in the variable control stage. This showed that the student was relatively 
intimidated or felt difficulties through verbal interaction. Thus, students should be provided with emotional support. 


Internal Experience 
Main Thought 


To analyze the internal experience of lower secondary school students, their main thoughts were presented at 
each stage of inquiry. The students’ responses were classified into experimental activities, emotions, interactions, 
and other sections (Table 5). In the observation stage, thoughts related to interaction had the highest frequency 
appearing 78 times (43.8%), followed by others (48 times, 27.0%), experimental activities (38 times, 21.3%), and 
emotions related to experimental activities (14 times, 7.9%). In terms of gender, male and female students thought 
about interaction the most, followed by thoughts related to others, experimental activities, and emotion sections. 
Specifically, in the interaction section of the observation stage, thoughts related to question (C1) had the high- 
est frequency. Meanwhile, in the experimental activities section, thoughts about result prediction (A6) had the 
highest frequency. However, there was no statistically significant difference by gender at the observation stage 
(p > .05): experimental activities (y’ = 3.638, df= 8, p = .888), interactions (y? = 4.570, df= 6, p = .600), and emotions 
(y? = .207, df= 1, p =.649). 
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Table 5 
Main Thoughts of Internal Experience at Each Stage of Inquiry 


Observation: n(%) Variable control: n(%) Conclusion drawing: n(%) 


Sections 
Male Female 5 a Male Female v Male Female v 
Al 2(2.2) 0(0.0) 3.638 6(6.6) 7(6.7) 3.853 0(0.0) 3(3.5) 10.413 
A2 4(4.5) 4(4.5) 4(1.1) 4(3.8) 7(9.6) 5(5.9) 
A3 4(4.5) 6(6.7) 0(0.0) 0(0.0) 2(2.7) 2(2.4) 
A4 4(1.1) 2(2.2) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 
Experimental _ A5 0(0.0) 0(0.0) 6(6.6) 13(12.5) 0(0.0) 2(2.4) 
activities AG 6(6.7) 8(9.0) 16(17.6) 13(12.5) 5(6.8) 17(20.0) 
) AT 0(0.0) 0(0.0) 0(0.0) 0(0.0) 1(1.4) 0(0.0) 
A8 0(0.0) 0(0.0) 0(0.0) 0(0.0) 2(2.7) 1(1.2) 
Ag 0(0.0) 1(1.1) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 
sub- —17(19.1) 21(23.6) 29(31.9) 37(35.6) 17(23.3) 30(35.3) 
total 
Bt 4(4.5) 8(9.0) 207 3(3.3) 4(3.8) 298 6(8.2) 8(9.4) 209 
Emotions B2 4(1.1) 4(1.1) 3(3.3 7(6.7 3(4.1) 6(7.1) 
(8) sub- _5(5.6) 9(10.1) 6(6.6) 11(10.6) 9(12.3)  14(16.5) 
total 
C1 27(30.3) 25(28.1) 4.570 —‘17(18.7) 13(12.5) 4.624 ~—-18(24.7) 6(7.1) 9.560 
C2 3(3.4) 2(2.2) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 
C3 3(3.4) 2(2.2) 7(7.7) 5(4.8) 3(4.1) 2(2.4) 
C4 4(1.1) 1(1.1) 3(3.3) 3(2.9) 0(0.0) 2(2.4) 
Interactions 
(Cc) C5 2(2.2) 2(2.2) 4(1.1) 5(4.8) 1(1.4) 3(3.5) 
(er) 2(2.2) 5(5.6) 5(5.5) 6(5.8) 3(4.1) 6(7.1) 
C7 3(3.4) 0(0.0) 0(0.0) 4(1.0) 0(0.0) 0(0.0) 
sub-  41(46.1) 37(41.6) 33(36.3) 33(31.7) 25(34.2) 19(22.4) 
total 
Others (D) 26(29.2) 22(24.7) 23(25.3) 23(22.1) 22(30.1) 22(25.9) 
aie 89 89 91 104 73 85 
(100.0) (100.0) (100.0) (100.0) (100.0) (100.0) 


In the variable control stage, thoughts related to experimental activities and interaction sections had the 
highest frequency. For male students, interaction was higher than experimental activity, but for female students, 
experimental activity was higher than interaction. However, there were no statistically significant differences in 
experimental activities (y? = 3.853, df= 8, p =.870), interactions (y? = 4.624, df=6, p = .593), and emotions (x? = .298, 
df = 1, p =.585). 

In the conclusion drawing stage, students mainly thought about the experimental activities section, followed 
by the interaction, others, and emotions sections. In terms of gender, male students thought about the interaction 
section the most, whereas female students thought about the experimental activities section the most. Specifically, 
in the experimental activities section, conducting an experiment (A2) had the highest frequency for male students, 
followed by result prediction (A6). By contrast, result prediction (A6) had the highest frequency for female students. 
In the interaction section, male and female students thought about question (C1) the most. However, there was 
no statistically significant difference by gender at the conclusion drawing stage (p>.05): experimental activities 
(y? = 10.413, df= 8, p = .237), interactions (y? = 9.560, df = 6, p = .144), and emotions (y? = .209, df = 1, p =.648). 

In the stage of inquiry activity, students thought about the experimental activities or interaction section the 
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most and the emotions section the least. There was a slight difference in the order of the frequency of main thoughts 
according to gender at each stage of inquiry activity, but there was no statistically significant difference (p>.05). In 
addition, it can be confirmed that lower secondary school students have thoughts unrelated to class considering 
the high frequency of the others section. This can be related to a previous study (Kwak et al., 2020) showing that 
concentration on class is lowered when there is no positive experience, which is a defining achievement in inquiry 
activities. Therefore, it is necessary to create an atmosphere in which lower secondary school students can enjoy 
inquiry so that they can be interested and focus on class and improve science positive experiences. 


Contribution of Actions 


The students’ responses to whether their actions during small-group science inquiry activities were helpful 
are as follows (Table 6). Most of the students thought that their actions helped their inquiry activities in all stages. 
However, 8 students in the observation stage, 13 students in the variable control stage, and 11 students in the 
conclusion drawing stage thought that their actions were not helpful to the inquiry activities. 


Table 6 
Contribution of Actions of Internal Experience at Each Step of Inquiry 


Observation: n(%) Variable control: n(%) Conclusion drawing: n(%) 


Male Female v Male Female v Male Female v 
Yes 36(0.9) 33(89.2) 33(82.5) 31(83.8) 33(82.5) 33(89.2) 
No 4(0.1) 4(10.8) 136 7(17.5) 6(16.2) 023 7(17.5) 4(10.8) 702 
Total 40(100.0) 37(100.0) 40(100.0) 37(100.0) 40(100.0) 37(100.0) 


Male and female students mostly thought that their actions were helpful in all stages of the experimental 
activities. However, among male students, 4 students in the observation stage and 7 students in the variable 
control and conclusion drawing stages thought that their actions were not helpful in the inquiry activities. In 
the case of female students, 4 students in the observation and conclusion drawing stages and 6 students in the 
variable control stage thought that their actions were not helpful to the inquiry activities. The contribution of 
actions for inquiry activities by gender is as follows: observation stage (y? = .136, p = .907), variable control stage 
(y? = .023, p = .881), and conclusion drawing stage (7? = .702, p =.402). There was no significant difference accord- 
ing to gender (p>.05). 

On the other hand, the results of reasons for the contribution of actions by students in inquiry activities are 
as follows (Table 7). Overall, regarding the reasons for responding that inquiry activities were helpful, the interac- 
tions section (54.3%) had the highest frequency, followed by experimental activities (36.4%) and emotions (8.7%) 
sections. Specifically, in the observation stage, the interactions section had the highest frequency, followed by 
the experimental activities and emotions sections. In the interactions section, behavioral participation (C5) and 
presentation of opinions (C3) had the highest frequency. In the experimental activities section, conducting ex- 
periments (A2) had the highest frequency. In the variable control stage, interaction had the highest frequency, 
followed by inquiry activities and emotions. Behavioral participation (C5), presentation of opinions (C3), and 
atmosphere (C6) had a higher frequency in the interaction section than in other inquiry activity stages. In the 
experimental activities section, experiment preparation (A1) and variable control (A5) had the highest frequency. 
In the conclusion drawing stage, interactions (54.4%) had the highest frequency, followed by inquiry activities 
(39.4%) and emotions (5.3%). Similar to other stages, behavioral participation (C5) and presentation of opinions 
(C3) had high frequency in the interactions section. Conducting experiments (A2) was presented most frequently 
in the experimental activities section. 

At each stage of the inquiry activity, students presented the reasons why their behavior contributed to the 
inquiry activity in the order of interactions, experimental activities, and emotions. However, the interaction was 
higher in the variable control stage than in the other stages. In the observation stage, more students responded to 
emotions (i.e., positive emotions) than in other stages. In the observation, variable control, and conclusion draw- 
ing stages, interactions had the highest frequency for males and females, followed by experimental activities and 
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emotions. There was no statistically significant difference according to gender (p > .05). In the observation stage, 
it was experimental activities (y? = 3.753, df = 8, p = .879), interactions (y? = 2.545, df = 6, p = .863), and emotions 
(y? = .000, df= 1, p =1.000). In the variable stage it was experimental activities (y? = 3.200, df= 8, p = .921), interac- 
tions (y? = 1.735, df = 6, p = .942), and emotions (x? = .000, df = 1, p =1.000). And in the conclusion drawing stage, 
it was experimental activities (y? = 3.346, df = 8, p = .911), interactions (y? = 2.811, df = 6, p = .832), and emotions 
(y? = .000, df = 1, p =1.000). 


Table 7 
Gender Differences in Reasons for the Contribution of Actions 


Observation: n(%) Variable control: n(%) Conclusion drawing: n(%) 
Sections 
Male Female al Male Female aa Male Female Vv 
Al 1(1.6) 1(1.8) 10(16.4) 4(7.4) 2(3.7) 3(5.0) 
A2 —17(27.4) ~—-13(23.6) 4(1.6) 2(3.7) 7(13.0) 11(18.3) 
A3 2(3.2) 2(3.6) 0(0.0) 0(0.0) 2(3.7) 2(3.3) 
Ad 1(1.6) 1(1.8) 0(0.0) 0(0.0) 2(3.7) 3(5.0) 
ae AS 1(1.6) 0(0.0) 9(14.8) 5(9.3) 0(0.0) 2(3.3) 
(A) AG 2(3.2) 0(0.0) 3.753 3(4.9) 3(5.6) 3.200 0(0.0) 2(3.3) 3.346 
AT 0(0.0) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 1(1.7) 
A8 0(0.0) 0(0.0) 0(0.0) 0(0.0) 1(1.9) 2(3.3) 
AQ 0(0.0) 1(1.8) 2(3.3) 0(0.0) 2(3.7) 3(5.0) 
24(38.7) — 18(32.7) 25(41.0)  14(25.9) 16(29.6) 29(48.3) 
Bt 11(17.7) 9(16.4) 0(0.0 4(7.4 5(9.3 1(1.7) 
— B2 0(0.0) 0(0.0) or 0(0.0) 0(0.0) 6b 0(0.0) 0(0.0) ar 
sub- (17 7) 9(16.4) 0(0.0) 4(7.4) 5(9.3) 1(1.7) 
total 
C1 0(0.0) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 0(0.0) 
C2 1(1.6) 1(1.8) 0(0.0) 1(1.9) 4(7.4) 2(3.3) 
C3 6(9.7) 8(14.5) 11(18.0) 10(18.5) 5(9.3) 4(6.7) 
C4 0(0.0) 2(3.6) 2(3.3) 4(7.4) 0(0.0) 0(0.0) 
Interactions 
() C5. 16(25.8) 14(25.5) 2.545 15(24.6) —«-14(25.9) 1.735 20(37.0) 22(36.7) «2.811 
C6 4(6.5) 3(5.5) 6(9.8) 6(11.1) 3(5.6) 1(1.7) 
C7 0(0.0) 0(0.0) 4(1.6) 1(1.9) 0(0.0) 1(1.7) 
27(43.5) —-28(50.9) 35(57.4)  36(66.7) 32(59.3)  30(50.0) 
oe 0(0.0) 0(0.0) _ 4(1.6) 0(0.0) _ 4(1.9) 0(0.0) 7 
ane 62 55 61 54 54 60 
(100.0) (100.0) = (100.0) (100.0) zi (100.0) (100.0) = 


Consciousness and Emotions 


Gender differences in consciousness and emotion felt by lower secondary school students during each in- 
quiry activity were analyzed (Table 8). Male students showed the highest score in “concentration” (4.18), followed 
by “teacher's guidance” (4.15), “help” (4.10), “interest” (4.08), and “effort” and “learning” (4.03). On the other hand, 
female students showed the highest score in “effort” and “teacher's guidance” (4.19), followed by “concentration” 
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(4.05), “interest” (3.89), and “help” (3.86). For male and female students, “teacher’s guidance,’ “interest,’ and “help” 
corresponded to the main consciousness felt during inquiry activities. 


Table 8 
Consciousness of Internal Experience during Inquiry Activities 


Consciousness Group N M SD t p 

ale 40 3.83 81 

Opportunity to choose 347 307 
Female 37 3.76 52 
ale 40 3.80 57 

Importance 304 362 
Female 37 3.73 92 
ale 40 4.08 84 

Interest 838 202 
Female 37 3.89 99 
ale 40 3.43 1.38 

Difficulty* -.141 444 
Female 37 3.46 92 
ale 40 3.85 1.00 

Enjoyment 677 .250 
Female 37 3.70 83 
ale 40 4.18 51 

Concentration 675 251 
Female 37 4.05 12 
ale 40 3.30 68 

High skill 638 .263 
Female 37 3.19 AQ 
i i ale 40 3.00 1.59 

Think of someting 1.645 052 
else Female 37 2.57 1.09 
ale 40 3.85 59 

Goal 1.018 156 
Female 37 3.65 .90 
ale 40 4.03 69 

Effort -.877 192 
Female 37 4.19 66 
ale 40 4.15 59 

Teacher's guidance -.222 A12 
Female 37 4.19 .60 
ale 40 3.90 86 

Fit together 406 343 
Female 37 3.81 1.00 
ale 40 3.68 84 

Reflection of ideas 122 452 
Female 37 3.65 96 
ale 40 4.03 69 

Learning .929 178 
Female 37 3.84 86 
ale 40 4.10 val 

Help 1.148 127 
Female 37 3.86 .90 
ale 40 3.81 95 

Total 800 215 
Female 37 3.70 98 


*negative item 


The mean for consciousness during inquiry activities was 3.81 for male students and 3.70 for female students, 
and there was no statistically significant difference between male and female students (p > .05). As such, the pres- 
ent study found no statistically significant difference regarding the specific consciousness for each item between 
male and female students. 

Table 9 shows the gender differences in emotion during inquiry activities. Emotion was measured using a 
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Likert scale using contrasting adjectives to express feelings or emotional states during the inquiry process. Male 
students felt “successful” the most with a score of 4.40, followed by “anxious” and “competitive” (4.36), “annoying” 
(4.32), and “pressured” (4.24). On the other hand, female students felt “annoying” and “competing” the most with a 
score of 4.26, followed by “anxiety” (4.12) and “pressured” (4.06). 


Table 9 
Emotions of Internal Experience during Inquiry Activities 


Emotions Group N M SD t p 
Male 40 3.35 1.07 
Joyful -.116 454 
Female 37 3.38 1.46 
Male 40 3.84 81 
Creative 573 284 
Female 37 3.71 1.06 
Male 40 4.27 93 
Pressured* 915 182 
Female 37 4.06 97 
Male 40 3.87 1.01 
Interesting 1.297 .100 
Female 37 3.53 1.35 
Male 40 3.92 85 
Boring* 1.555 062 
Female 37 3.53 1.35 
Male 40 4.35 .90 
Anxious* 905 184 
Female 37 4.12 1.44 
Male 40 4.27 1.09 
Annoying* 023 491 
Female 37 4.26 1.05 
Male 40 4.32 18 
Competitive* 257 399 
Female 37 4.26 1.11 
Male 40 3.81 1.16 
Active 478 317 
Female 37 3.68 1.62 
Male 40 3.84 92 
Curious 171 432 
Female 37 3.79 1.38 
Male 40 3.89 1.10 
Cooperate 519 303 
Female 37 3.77 1.03 
Male 40 3.73 1.20 
Involved 1.045 .150 
Female 37 3.47 98 
Male 40 4.24 97 
Successful 1.811 .037* 
Female 37 3.82 94 
Male 40 3.98 1.02 
Total 1.538 .069 
Female 37 3.80 1.12 


*negative item 


Male students felt “successful” the most during small-group inquiry activities, which was statistically signifi- 
cant compared with female students (p < .05). However, it was confirmed that lower secondary school students, 
regardless of gender, felt generally negative feelings such as anxiety or competition while conducting small-group 
inquiry activities (p > .05). Therefore, it is necessary to make efforts to reduce anxiety and negative mood that lower 
secondary school students feel in science inquiry activities. 
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Discussion 


Most of the external experiences of grade 6 students in the Republic of Korea during inquiry activities were 
related to class activities, but there were many internal experiences unrelated to inquiry activities. In the stage of 
inquiry activity, approximately 1% of other actions and thoughts were unrelated to the inquiry activity, including 
personal actions, small talk, game thoughts, and break time thoughts, which are external actions (Table 4). On the 
other hand, the frequency of others (D) in internal experience was 23%-27% (Table 5). This means that students 
experience the inquiry in action but think about thoughts that are unrelated to the inquiry. In other words, about 
1/4 of all students are unable to concentrate during small-group inquiry activities and engage in small talk, jokes, 
or personal actions and thoughts that are unrelated to the inquiry activities. These results indicate that many lower 
secondary school students feel that the inquiry activities are not enjoyable or interesting. Comparing this finding 
to the results of Kim and Yang (2005), inquiry activities can act as one of the causes of disliking science. 

In terms of external experience, students mainly conducted experiments with peers or conversed with peers 
about experiments rather than listening to the teacher's directives. nevertheless, approximately 20% of the exter- 
nal experiences experienced by students listened to the teacher's directives as the main action. This means that 
the main focus is on the inquiry under the directive of the teacher, rather than on the self-directed inquiry by the 
student. Therefore, a method to guide students to self-directed inquiry activities is required. In addition, verbal 
interaction (i.e., conversation with peers about experiments) frequently occurred in the variable control stage. 
Consistent with previous studies (Yu & Choi, 2012), students’ spontaneity and high social interaction are required 
in the variable control and experimental design stages. Accordingly, it could be inferred that this was because one 
of the reasons for the low verbal interaction of students during inquiry activities was that they did not proceed 
with the variable control stage (Kim & Kim, 2012; Kwon & Kim, 2016). 

In the results of the main actions of external experiences at each stage of inquiry, students in the Republic of 
Korea have a low level of variable control ability formation; in the case of lower secondary school students, many lack 
variable control ability (Kim & Kim, 2012). Thus, there is a high possibility of feeling negative emotions. The lack of 
awareness of variables can act as a cause of experimental performance failure (Germann et al., 1996). Since students 
can learn the contents related to simple variable control and training is effective (Lawson & Wollman, 2003), it is 
necessary to provide and implement a program for the variable control stage for lower secondary school students. 

On the other hand, it was confirmed that students did not perform activities related to conclusion drawing, such 
as conducting experiments or conducting observations, in the conclusion drawing stage. This finding is consistent 
with that of Lim et al. (2011), who found that lower secondary school students have difficulties in interpreting data 
and drawing conclusions. Data interpretation is a high-level inquiry function and is a scientific literacy required of 
students in relation to the rapid increase in the amount of information in modern society (Gotwals, 2006). Moreover, 
data interpretation must be preceded in order to draw conclusions (Lim et al., 2011). Therefore, it is necessary to 
support activities such as cultivating the ability to identify the relationship between two variables (Kanari & Mil- 
lar, 2004) or enhancing data interpretation skills through experience in graph interpretation (Choi et al., 2001) for 
lower secondary school students to perform data interpretation and conclusion drawing. 

Students were engaged in activities that did not correspond to the inquiry stage. For example, in the observa- 
tion stage, there was a student who prepared an experiment, conducted an experiment, and predicted the result of 
the experiment. There were students who did not design the experiment, but rather prepared and performed the 
experiment in the variable control stage and made predictions in the conclusion drawing stage. This is because stu- 
dents do not clearly distinguish which activities need to be or do not need to be performed at each stage of inquiry. 

Students thought that their activities were helpful to their peers in inquiry activities. This can be inferred from 
the result that the main activity is the progress of the experiment with the peers or the conversation about the 
experiment accounted for more than 70% (Table 3). However, more interaction took place in the variable control 
stage than in the other exploration stages (Table 4). In other words, a variable control step should be included to 
increase the interaction between students during inquiry activities. 

As the result of internal experiences, “teacher's guidance’, “interest”, and “help” were included regardless of 
gender in the consciousness and emotions. This is in line with the results of previous studies (Hofstein & Lunetta, 
2004; Yang et al., 2006) that inquiry activities help increase interest in and understanding of science. 

Students often felt negative emotions, including anxiety, during the inquiry activities regardless of gender. 
This is consistent with the results of previous studies on scientific anxiety (Jeong & Kim, 2011; Kim et al, 2014). 
Positive emotions lead to emotional and psychological actions than negative emotions and act as factors that 
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increase patience and creativity, giving individuals the strength to adapt well to new and unfamiliar experiences 
(Ko, 2016). Also, the lower the level of negative emotions such as anxiety or aggression in the small-group inquiry 
activity, the more positive the perception of the small-group inquiry activity, and the more active the interaction 
(Yeo & Kim, 2005). Furthermore, it is necessary to make efforts to reduce anxiety and negative mood that lower 
secondary school students feel in science inquiry activities. In particular, female students were relatively less likely 
to feel more successful than male students. Therefore, it is necessary to find a way to provide female students with 
the experience of success in inquiry activities. In addition, it is very important to help lower secondary school stu- 
dents adapt to new and unfamiliar topics through scientific inquiry activities and have positive emotions about 
the inquiry activities in order to perform well. 


Conclusions and Implications 


This study analyzed the external and internal experiences experienced by lower secondary school students 
at each stage during small-group science inquiry activities. First, there was no difference in the external and inter- 
nal experiences of male and female students in each stage of the inquiry activity. In addition, the main activities 
performed in the process of inquiry activities were activities related to colleagues and experiments. 

Second, there were many cases during inquiry activities in which activities in other stages of inquiry were 
carried out in addition to the activities that were mainly performed in the corresponding stage. For example, 
students answered that conducting the experiment was the main action performed in the observation stage, and 
experiment preparation was performed more than experiment design in the variable control stage. In addition, 
students responded that they did more experiment preparation, conducting experiments, and observation activi- 
ties in the conclusion drawing stage (Table 4). 

Third, approximately 27% of students said that they had internal experiences unrelated to inquiry during inquiry 
activities. Also, some students thought that their main actions were not helpful to the inquiry activities. In particu- 
lar, this was relatively more common in the variable control stage and conclusion drawing stage, which requires 
a high-level inquiry function, compared with the observation stage, which requires a simple inquiry function. It is 
considered that there are many cases in which students are unable to participate in inquiry activities that require 
high-level inquiry skills such as designing and interpreting experiments and interaction with peers. Therefore, it is 
required to guide all members in the process of inquiry so that they can participate in inquiry activities. Through 
this, it is hoped that students will become interested in science through inquiry activities. 

Fourth, there was little difference between genders in the internal and external experiences experienced in 
the process of conducting scientific inquiry, but female students were found to have relatively fewer internal ex- 
periences that made them feel more successful than male students. This may lead to differences in science-related 
attitudes between male and female. 

Based on the results, it is suggested to include a variable control step in designing the experiment to allow 
students to actively interact during inquiry activities. However, some differences emerged in attitudes toward 
science, such as feeling successful about science. This difference in attitudes toward science despite the same ex- 
perience can adversely affect the success of scientific inquiry. Therefore, it is necessary to provide female students 
with a successful internal experience to increase their confidence and attitude toward science, while equal inquiry 
activities that do not discriminate according to gender are carried out. Moreover, as 20% of the students consid- 
ered listening to the teacher's directive as their main activity, it is expected that the inquiry activity will become a 
student-centered inquiry activity. Since this study analyzed experiences in open-ended inquiry, it is expected that 
the analysis will be conducted in various forms of inquiry, such as guided inquiry. 
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Abstract. The social responsibility of 
college students in the later stages of 
adolescence for sustainable development 
is emphasized, and the role of universities 
has become a crucial task. This study 
aimed to explore the level of college 
students’ cognition, attitude, and behavior 
towards sustainable development and 

the association among them through 

the moderation effect of informal online 
learning. The difference verification 

and structural equation modeling were 
applied to 559 students who majored 

in science, and statistical validity was 
confirmed. Regarding the sustainable 
development level of college students, there 
was a significant difference in cognition, 
attitude, and behavior towards sustainable 
development between males and females, 
formal education, and informal online 
learning. Further, college students’ attitudes 
towards sustainable development played 

a significant mediating role between 
cognition and behavior. The study also 
found a significant moderation effect 

of informal online learning on students’ 
cognition and attitude toward sustainable 
development behavior. Accordingly, this 
study suggests that universities should 
systematically design educational programs 
to promote college students’ behavioral 
change toward sustainable development 
through personal life-oriented learning 
courses. 
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Qi Liu, 

Xiaoxia Tian, 
Younghwan Bang, 
Kyung Hee Park 


Introduction 


As society continues to prioritize rapid economic development and 
growth, excessive development, and consumption have led to the depletion 
and degradation of resources, environmental destruction and pollution, and 
global warming. These ecological problems have become important global 
issues (Andersen et al., 2018; Choi et al., 2010; UNESCO, 2005). To address 
these issues, it is necessary to strengthen education for awareness and un- 
derstanding of sustainable development. Especially for college students, as 
the backbone of future society, their cognition and behavior have a crucial 
impact on the sustainable development of future society (Avila et al., 2019; 
Wals & Jickling, 2002; Walshe, 2017). 

Many universities offer courses on sustainable development to educate 
students on how to contribute to society and the environment (Tilbury, 2004; 
Veidemane, 2022). The significance of conducting sustainable development 
education in universities lies in the fact that the college's responsibility 
extends beyond imparting knowledge, as it also involves cultivating the 
personality and values of its students. 

In addition, as part of society, college students have a direct impact on 
the environment and society through their behavior and habits. Providing 
sustainable development education and training for college students can 
cultivate their environmental awareness and sustainable development con- 
cepts, guiding them to adopt sustainable lifestyles and make contributions to 
the future sustainable development of society (Min, 2019; Shephard, 2008). 

Furthermore, the current use of digital media is recognized as an impor- 
tant communication tool, and it is attracting attention as an optimal tool for 
realizing social learning as it enables knowledge and information sharing, 
dissemination, communication, and collaboration (Lim et al., 2012). College 
students obtain information related to sustainable development through 
mass media and provide opportunities for activities related to sustainable 
development through methods such as online forums (Hori & Fujii, 2021). 
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With this social trend, personal learning using information and communication technology (ICT) is increasing and 
evolving as a tool for informal learning (Kim & Yi, 2011). 

Nevertheless, sustainable development is still mostly at a theoretical level, and students’ perceptions and ac- 
tions toward the environment need to match the current environmental problems (Shephard, 2008; Vandemoortele, 
2018). Most previous studies have tended to separate the cognition, attitude, and behavior of sustainable develop- 
ment (Michalos et al., 2011; Min, 2019; Torbjérnssona, 2011), and empirical studies on the association between these 
factors have been limited. In this context, this study explores the association between personal characteristics and 
factors influencing sustainable development to enhance the sustainable development behavior of college students. 


Literature Review 
College Students’ Sustainable Development 


Ever since the World Commission on Environment and Development (WCED) defined sustainable develop- 
ment in 1987 as‘meeting the needs of the present without compromising the ability of future generations to meet 
their own needs; the complexity of sustainable development due to its characteristics, there have been various 
interpretations and criticisms in its definition. Among them, this study conceptualizes sustainable development as 
the comprehensive development strategy for the 21st century that emphasizes the integration and balance among 
the environment, economy, and society, based on the global agenda of the UN (2015) and the Triple Bottom Line 
theory (TBL theory) of Elkington and Rowlands (1997). 

In particular, the Chinese government and the Ministry of Education recognize the importance of sustainable 
development as a national and global agenda and are implementing a series of policies and measures to promote 
the sustainable development of university students. In this regard, in 2019, the China Center for Environmental 
Protection and Publicity and Education issued the ‘Guidance Opinion on Strengthening the Construction of Ecological 
Civilization in Higher Education’ to strengthen ecological civilization education in universities and cultivate students’ 
awareness of environmental protection. It has been announced that university education will promote the con- 
struction of an ecological civilization. 

In addition, in China, influenced by the Rio Conference in 1992, education for sustainable development tends 
to be discussed centering on environmental education (Qu, 1993), and education for sustainable development for 
college students also emphasizes environmental protection and resource conservation (Pan, 2015). Recently, in 
order to promote a sustainable development policy called ‘Green Walk’, sustainable development using eco-friendly 
transportation is supported (Chen, 2014; Wang et al., 2018). For example, there are attempts by college students 
to reduce resource waste and carbon emissions by using new transportation methods such as shared bicycles and 
carpools (Shen & Liu, 2017; Wang et al., 2018). 

Nevertheless, there are still some limitations in the cultivation of sustainable development among Chinese 
college students. First, many college students have received less education on sustainable development due to the 
lack of curriculum and educational resources (Huang, 2014; Yang et al., 2017). Second, as some college students 
lack in-depth thinking about sustainable development, they tend to underestimate the value of the environment 
and resources and lack action and initiative to promote sustainable development (Hu, 2013; Zhang, 2020). In 
addition, studies that can further strengthen the combination of theory and practice in the field of sustainable 
development to improve the practical ability of college students for sustainable development are needed (Yang 
et al., 2017; Yi & Lu, 2022; Yu & Yang, 2019). 

Accordingly, it is important for all stakeholders, including government, civil society organizations, corporations, 
and universities, to collaboratively design and implement practical plans that incorporate sustainable development 
principles and visions. Additionally, educating individuals about sustainable development should be a priority 
(Hesselink, 2000; UNESCO, 2005). 


Factors Influencing Sustainable Development 
Sustainable development has become a critical issue in higher education institutions worldwide, as they are 
not only responsible for equipping students with knowledge and skills to address global sustainability challenges 


but also must lead by example and implement sustainable practices themselves. However, implementing sus- 
tainable practices in higher education institutions has its challenges (Tilbury, 2004; Veidemane, 2022). According 
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to Avila et al. (2019), higher education institutions around the globe face a multitude of challenges that hinder 
students’ promotion of sustainable development, and this is equally true for Asia. Among those challenges, this 
study explored the structural association between sustainable development cognition, attitude, and behavior 
among college students. 

Firstly, Michalos et al. (2011) argued that the association between cognition and attitude toward sustain- 
able development is stronger than other associations and that the association between cognition and behavior 
is also significant. It has been shown that there is a statistical correlation between cognition and behavior (Yoo 
& Lee, 2014), and that there is also a significant correlation between attitude and behavior (Min, 2019; Walshe, 
2017). In addition, studies conducted by Yoo and Lee (2014) and Yoo and Oh (2016) provided evidence sup- 
porting the view that cognition has a substantive impact on behavior. The research conducted by Yoo and Oh 
(2016) also suggested that attitudes play an important role in shaping individual behavior. In other words, even 
in the absence of external pressure or incentives, an individual’s views on a particular issue or behavior strongly 
influence their actions. 

When considering previous studies, one thing to consider in studies of college students’ cognition and related 
behaviors on sustainable development is whether there is a difference in gender, formal education (sustainable 
development learning experience in college) and informal online learning (using ICT for sustainable development). 

Recent studies have shown that females are more interested in the environment and more sensitive to envi- 
ronmental issues than males, resulting in a significant difference in females’ cognition and behaviors for sustainable 
development compared to males (Min, 2019). Furthermore, previous studies have demonstrated that individual 
value systems can also play a significant role in shaping attitudes towards sustainable development. For instance, 
individuals who prioritize human-centered values, such as care for others and social justice, tend to have more 
positive attitudes towards sustainable development than those who prioritize self-enhancement values, such as 
achievement and power (Torbjérnssona, 2011). 

In addition, classes related to sustainable development at school can help college students understand the 
concepts and principles of sustainable development, and through these classes, college students can understand 
the meaning and background of sustainable development, thereby contributing to raising awareness of sustainable 
development (Mahtab & Asghar, 2021). Similarly, Janmaimool and Khajohnmanee (2019) found that students with 
a learning experience in sustainable development have more positive attitudes than students without a learning 
experience and related to sustainable development in school can promote positive attitudes, and the attitude 
influences environmental behavior. 

Hori and Fujii (2021) suggested that due to the rapid development of ICT, an increasing number of students 
are learning about sustainable development through the Internet or mobile phones and that college students 
actively collect information related to sustainable development. It can positively influence the values of people to 
support sustainable development and take more action on sustainable development (Gu, 2022). 

Understanding these associations and differences, this study can provide insights into how college can better 
educate and engage students on sustainable development, and how individuals can be encouraged to adopt more 
sustainable behaviors in their daily lives. Therefore, the following three research questions were posed: 

1. Are there differences in cognition, attitude, and behavior of sustainable development among college 
students? 

2. Is there a mediating effect of attitude between cognition and behavior of sustainable development? 

3. Doinfluences of sustainable development attitudes on behavior differ according to gender and learn- 
ing experiences? 


To address these questions, the study formulated an analytical model, using “the cognition of sustainable 
development” as the independent variable, “attitude” as the mediating variable, and “behavior” as the dependent 
variable. The association between the variables is illustrated in Figure 1. Building on the prior studies, this study 
constructed both partial and full mediating analytical models, considering the findings of some studies that indicate 
no direct effect of the cognition of sustainable development on behavior. Min (2019) suggested that an association 
that could affect behavior is inconclusive in a state where the knowledge level of sustainable development is low. 
This may be because sustainable development cognition is transferred to action only when it is internalized into 
an attitude. Therefore, this study also included a full mediating model (Figure 2) that suggests that cognition does 
not have a direct impact on behavior. Finally, in order to further verify the model validity, this study considered the 
moderation effect on the structural model of gender, formal education experiences, and informal online learning 
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related to sustainable development (Figure 3). By verifying the statistical significance of the moderation effect, the 
study intended to enhance the explanatory and predictive power of the research model. 


Figure 1 Figure 2 Figure 3 
Partial Mediating Model Full Mediating Model Meditation Model by Multi-group 


Gender Formal education, 
Online-informat 
SD-Attitude SD-Attitude ine-nformal leering 


) 


SD-Attitude 


SD-Behavior SD-Cognition SD-Behavior SD-Behavior 


SD-Cognition 


Naa? 


SD-Cognition 


Research Methodology 


Research Design 


This study is quantitative research aimed at examining the levels of college students’ cognition, attitude, and 
behavior towards sustainable development and the associations among these variables. A survey questionnaire 
was administered to undergraduate students from a comprehensive university in China, and a structural equation 
modeling approach was used to analyze the data. 

The survey was conducted for one month from May to June 2022. The questionnaire was distributed during 
the night self-study time of the students, and before completing the questionnaire, the students read and signed 
the informed consent form that explained the purpose of the study and the confidentiality of the survey responses. 


Participants 


Study participants are recruited from the only four-year university in C City, a city abundant in natural re- 
sources. The university has established departments related to sustainable development and carries out related 
activities to promote cognition, attitude, and behavior of sustainable development among students majoring 
in science-related fields. In this study, a sample was chosen from this university to obtain representative data on 
sustainable development. Students enrolled in grades 1 to 4 were included in the sample to reflect the various 
levels of sustainable development. There were slightly more male students (308, 55.1%) than female students (251, 
44.9%). By grade, the sample consists of 169 first-year students (30.2%), 134 second-year students (24.0%), 125 
3rd-year students, and 131 4th-year students (24.4%). Finally, the study analyzed the distribution of students with 
prior formal educational experience and informal online learning. Of the students surveyed, 216 students (38.6%) 
had prior formal educational experience, while 343 (61.4%) students did not have such experience. Regarding 
informal online learning, 310 students (55.5%) indicated a higher level of informal online learning than the overall 
average, while the rest, 249 students (44.5%) had a lower-than-average level. 

This study collected 580 questionnaires, of which 559 copies were put into the analysis, excluding 21 ques- 
tionnaires that were answered insincerely. Although the sample size is small, this study adopted a rigorous sample 
selection process and ensured the representativeness and diversity of the sample. 


Measurements 


The study adopts a measure of cognition, attitude, and behavior of sustainable development targeting young 
people, developed by Michaelos et al. (2011). This measurement tool is considered reliable and consistent: the 
overall reliability of the cognition tool is Cronbach's a = 8.790, the attitude tool is Cronbach's a = 0.770, and the 
behavior tool is Cronbach's a = 0.630. This measurement tool consists of 47 items, but this study uses items found 
to be reliable, removing the items with values less than 0.6 on the reliability index. The measurement tool used in 
this study consists of a total of 24 items, comprised of 9 items for cognition, 5 items for attitude, and 10 items for 
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behavior. To ensure the content validity of the measurements, three education experts reviewed the contents of 
each item and confirmed the suitability of the final 24 items. Each item is reported on a Likert 5-point scale (1 = 
not at all; 5 = very much), where higher scores signify higher levels of awareness, attitude, and practice of sustain- 
able development. 

To verify the validity of the measurement tools, exploratory factor analysis (EFA) and confirmatory factor 
analysis (CFA) were conducted (shown in Table 1). In the process of exploratory factor analysis, principal compo- 
nent analysis was performed to extract three factors. Twenty-four items with factor loadings exceeding 0.5 and no 
cross-loading were maintained. The goodness of fit of the KMO sample is judged to be acceptable when it is 0.6 
to 0.7 or higher, and the factor analysis result is 0.966, and in Bartlett's sphericity test, the significance probability 
is 0.000 and the significance level p < 0.001, and the model is confirmed to be suitable. Next, CFA was performed 
to evaluate the structure of factors derived by EFA. The model fit index was ¥7/df = 2.284 (< 3), TLI = 0.960 (> 0.9), 
CFI = 0.964 (> 0.9), and RSMEA = 0.048 (< 0.05), confirming that the model was appropriate. In addition, in CFA, 
the factor loading is between 0.550 and 0.780, the average variance extraction (AVE) value is between 0.467 and 
0.565, and the combined reliability (CR) value is between 0.867 and 0.905. The measurement tool of this study was 
found to be appropriate. 

The Cronbach's alpha values of the last question on the survey for the cognition, attitude, and behavior mea- 
surements were 0.928, 0.898, and 0.917 respectively, which met the acceptance standard of over .6 for reliability. 
This study performed confirmatory factor analysis to evaluate the fit of the measurement model. 


Table 1 
Validities of the Sustainable Development Items 


Items Mean EFA CFA 


$D-Cognition (Cronbach’s a 0.928, CR 0.905, AVE 0.514) 


C1. Economic development, social development and environmental protection are all needed for sustainable 3.75 0.651 0.698 


development 
C3. Sustainable development is about both future needs and what we need today 4.00 0.708 0.741 
C5. China’s overall use of energy remains to be improved 3.93 0.778 0.849 


C9. Sustainable development tries to balance human and economic success with cultural traditions and respect 


for the earth’s natural resources aes oe ics 


C10. We can slow the rate of climate change 3.53 0.692 0.751 
C12. Conservation of fresh water is not important in China because we have plenty 3.69 0.717 0.792 
C13. Maintaining biodiversity is essential to the health of ecosystems 3.64 0.738 0.802 


C15. Non-renewable resources should not be used faster than the rate at which we find substitutes that are 


3.71 0.666 0.755 
renewable 


C16. It is useful to estimate the monetary value of the services that the ecosystem provides to us 3.69 0.731 0.772 


SD-Attitude (Cronbach’s a 0.898, CR 0.867, AVE 0.565) 


A3. Companies should try to avoid making design disposable products 3.57 0.764 0.831 
Ad. Laws and rules to protect the environment need to be more strict than they are now 3.29 0.751 0.793 
A8. Companies that are environmentally responsible are more likely to make a profit over the long run 3.57 0.755 0.832 
A10. Governments should encourage greater use of fuel-efficient vehicles 3.62 0.780 0.704 
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Items Mean EFA CFA 
A113. Taxes on polluters should be increased to pay for damage to communities and the environment 2.67 0.707 0.837 


SD- Behavior (Cronbach's a 0.917, CR 0.897, AVE 0.467) 


B1. | walk or bike to places instead of going by car 3.89 0.739 0.779 
B2. | use reusable containers in order to reduce waste 2.85 0.669 0.69 
B3. At home, | try to recycle as much as | can 3.23 0.622 0.707 
B5d. | have taken a course in which we talked about sustainable development 3.36 0.703 0.799 
B7. | pick up litter when | see it in a park or a natural area 2.53 0.664 0.712 
B8. | often look for signs of damage to our environment 2.53 0.750 0.830 
B10. | sort the trash before | throw it away 3.32 0.665 0.687 
on : BL ee purchasing goods from companies with a poor track record on caring about their workers or the 4.09 0.705 0.784 
B13. | have changed my personal lifestyle to reduce waste 3.03 0.550 0.675 
B14. | have studied some issues related to climate change 2.90 0.744 0.759 


Data Analysis 


To ensure the quality and reliability of the collected data, several statistical analyses were conducted. Firstly, 
descriptive statistical analysis was conducted to obtain the basic information of the data, such as the mean, stan- 
dard deviation, and range of each variable. In addition, Cronbach's alpha coefficient was calculated to measure the 
internal consistency of the items and ensure the reliability of the measurement tool. Moreover, the questionnaire 
was translated back into Chinese, and CFA was conducted for the validity of the measurement tool. In this study, 
the hypothesized latent constructs were cognition, attitude and behavior of sustainable development, and the ob- 
served variables were the items in the questionnaire. Furthermore, the study explored the differences in sustainable 
development according to personal background variables such as gender, formal education, and informal online 
learning by conducting independent t-tests. By examining the differences in sustainable development between 
different groups, the study can gain insights into how personal background variables affect sustainable develop- 
ment behaviors. To explore the association between cognition, attitude and behavior of sustainable development, 
a structural equation model (SEM) was constructed, and the mediating effect of attitude was verified. Finally, a 
multi-group analysis was conducted to verify the moderation effects of gender, university class experience related 
to sustainable development, and informal online learning. All of the aforementioned analyses were conducted 
using SPSS 26.0 and AMOS 26.0 programs, which are commonly used statistical analysis software. 


Research Results 
Differences in Sustainable Development Behavior 
In order to understand the level of cognition, attitude, and behavior of university students on sustainable 
development, the study examined the differences by gender, class experience, and ICT activity (shown in Table 2). 


Overall, the behavior of sustainable development (M = 3.17) was the lowest. 
The disaggregated analyses show that there are statistically significant differences in cognition (t = -3.14), 
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attitude (t = -2.72), and behavior (t = -2.14) of sustainable development by gender. The findings indicate a signifi- 
cant difference by gender, with higher levels of cognition, attitude, and behavior of sustainable development of 
female students than those of male students. With regard to SD-related formal education experience, there were 
statistically significant differences in cognition (t = 8.40), attitude (t = 4.88), and behavior (t = 7.62) about sustain- 
able development. It was confirmed that students with experience taking SD-related classes have higher levels of 
cognition, attitude, and behavior than students without class experience. With regard to informal online learning, 
the study found a statistically significant difference in the cognition of sustainable development (t = -9.44), attitude 
(t=-7.79), and behavior (t= -9.02). The findings confirm that the students with informal online learning have higher 
levels of sustainable development. 


Table 2 
Comparing the Differences based on Personal Characteristics 


SD-Cognition SD-Attitude SD-Behavior 
Distinction 
M(SD) t M(SD) t M(SD) t 

Total 3.82(0.69) - 3.34(0.96) - 3.17(0.74) 
Male 3.74(0.68) 3.24(0.97) 3.11(0.74) 

Gender -3.14** -2.72* = =——___—_____ -2.14* 
Female 3.93(0.69) 3.46(0.93) 3.25(0.74) 
i Yes 4.11(0.53 3.59(0.89 3.46(0.70 

Formal education (0.53) ne ) jae (0.70) aren 
experience No 3.64(0.72) 3.19(0.96) 2.99(0.71) 
High level 4.06(0.53) 3.61(0.83) 3.41(0.70) 

Informal online learning 9.44*** 7.79*** 9.02*** 
Low level 3.54(0.76) 3.01(0.99) 2.88(0.69) 


Note. *p < .05, **p < .01, ***p <.001 


Verification of the Mediating Effect of Sustainable Development Attitude 


In order to further analyze the association among sustainable development cognition, attitude and behavior, 
this study set the attitude towards sustainable development as the mediating variable and compared the fit of 
the partial mediating model and the full mediating model as shown in Table 3. Both models met the criteria of be- 
ing less than .08 for root mean square error of approximation (RMSEA) and higher than .90 for Tucker Lewis index 
(TLI) and comparative fit index (CFI). The difference in chi-square values between the partial mediating model 
and the full mediating model was 125.767, which was significant as it was larger than the threshold of 3.84 for the 
difference of 1 degree of freedom. However, the chi-square test can only verify whether the model is significant 
but cannot determine which model is better. Therefore, other indicators, such as the Akaike information criterion 
(AIC) were needed to evaluate the models. The smaller the AIC value, the more simplified and the better fit the 
model (Akaike, 1974). The full mediating model has the smaller AIC value in this study, so it was determined to be 
the more suitable model. 


Table 3 
The Fitness of the Mediating Research Model 


Model df lf TLI CFI RSMEA AIC 
Partial Mediating 250 2.778"* 0.945 0.950 0.056 842.381 
Full Mediating 249 2.284*** 0.960 0.964 0.048 718.614 


Note. ***p < 0.001 


The result of the path coefficient analysis in the final model, which is the full mediation model, as seen in 
Figure 4, showed that all path coefficients were significant at p < 0.001. It indicates that cognition of sustainable 
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development has a positive effect on attitudes (B = 0.74, p < 0.001) and attitudes have a positive effect on behaviors 
(8 = 0.76, p < 0.001). This suggests that as students’ cognition of sustainable development increases, their attitudes 
towards sustainable development also strengthen. In turn, as their attitudes towards sustainable development 
strengthen, their sustainable behaviors also increase. These findings provide valuable insights for educators and 
policymakers to develop effective strategies to promote sustainable development among college students. 

This study used bootstrapping to verify the mediation effect, and the sampling time was 2000. As shown in 
Table 4, the mediating variable, attitude, had a significant effect between cognition and behavior, and the indirect 
effect value was 0.556 (Bias-corrected 95% C/ = 0.490 ~ 0.616). The confidence interval did not include 0 and the 
p-value was less than .001, which confirmed that attitude had a mediating effect. Therefore, the results highlight 
the importance of promoting positive attitudes towards sustainability to encourage sustainable behavior among 
college students. 


Figure 4 
Path Diagram and Standardized Estimate of Research Model 
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Table 4 
Significance Verification of Mediating Effect 


P Bootstrap 2000 times 
Paul eaulnate oe 95% bias-corrected Cl 
Cognition — Attitude — Behavior 0.556*** 0.044 0.490 ~ 0.616 


Note. ***p < .001 


Verification of the Multi-group Comparison 


Finally, a multi-group analysis was conducted to verify the moderation effect of gender, formal education 
(sustainable development learning experience in college), and informal online learning (using ICT for sustainable 
development). A measurement equality test was conducted to check whether the variables were equally recognized 
according to gender, formal education, and informal online learning. It was confirmed that the goodness-of-fit 
indices of all models were acceptable, as shown in Table 5. 

The model fit according to gender confirmed that ’/df = 2.640 (< 3), CFl = 0.901 (> 0.9), and RMSEA = 0.054 
(< 0.08), indicating an acceptable fit. The model fit according to educational experience showed that ¥7/df = 2.648 
(< 3), CFl = 0.890, and RMSEA = 0.054 (< 0.08). The model fit according to informal online learning revealed that 
?/df = 2.875 (< 3), CFl =.871, and RMSEA = 0.058 (< 0.08). Although the CFI value is slightly lower than 0.9, con- 
sidering the other goodness-of-fit indices, the model was deemed acceptable. 
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Table 5 
Nested Multi-group Model Fits 


Structural residuals Cldf CFI RMSEA 
Gender 2.640*** 0.901 0.054 
Formal education experience 2.648*** 0.890 0.054 
Informal online learning 2.875*** 0.871 0.058 


Note. df = 573, ***p < .001 


The findings showed that the sustainable development cognition of the two groups by gender had a posi- 
tive (+) significant effect on attitude (p < .001), and attitude had a positive (+) significant effect on behavior. (p < 
.001). Since the difference between ‘cognition — attitude path’ between male and female groups is .794, and the 
difference between ‘attitude — behavior path’ 1.616 is smaller than |1.96|, the difference in path is not statistically 
significant. As a result, the findings indicate no difference in the effect of attitudes toward sustainable development 
on practice according to gender. That is, the moderation effect of gender was not significant. 

The findings showed that the cognition of sustainable development of the two groups had a positive (+) 
significant effect on their attitude (p < .001), and attitude had a positive (+) effect on behavior. It was found to 
have a statistically significant effect (p < .001). Since the difference in ‘cognition — attitude path’ between groups 
with or without class experience is 0.659, and the difference in ‘attitude — behavior path’ -1.477 is smaller than 
|1.96], the difference in path is not statistically significant. Accordingly, the moderation effect of class experience 
at university was not significant. 

The findings also reveal that the cognition of sustainable development of the two groups according to in- 
formal online learning has a positive (+) significant effect on attitude (jp < 0.001), and attitude had a positive (+) 
significant effect on behavior (p < .001). The difference in the ‘cognition — attitude path’ between the below and 
above-average groups in informal online learning was 1.963, and the difference in the ‘attitude — behavior path, 
2.365, was greater than |1.96], so the difference in the path was statistically significant. Accordingly, it was found 
that there was a difference in the effect of attitudes toward sustainable development on behavior according to 
informal online learning. In other words, the moderation effect of informal online learning was significant in the 
structural relationship of sustainable development (shown in Table 6). 


Table 6 
Significance Verification of Multi-group Path Analysis 


Male Female 


Path toward gender Benwern 
B SE p SE groups 
Cognition — Attitude 0.712*** 0.084 0.737*** 0.100 0.794 
Attitude + Behavior 0.699*** 0.044 0.819*** 0.051 1.616 
Yes No 
Path toward formal education Perse) 
p SE p SE groups 
Cognition — Attitude 0.655*** 0.113 0.739*** 0.086 0.659 
Attitude — Behavior 0.756*** 0.059 0.726*** 0.040 -1.477 
High level Low level 
Path toward informal online learning penveen 
B SE B SE groups 
Cognition — Attitude 0.709*** 0.116 0.673*** 0.094 1.963 
Attitude — Behavior 0.759*** 0.052 0.675*** 0.046 2.365 


Note. ***p < .001 
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Discussion 


Sustainable Development Level and Differences 


As the world grapples with climate change, resource depletion, and social inequality, sustainable de- 
velopment has become imperative for college students expected to uphold social responsibility. However, 
despite the growing cognition and acceptance of sustainable development among students, the current study 
showed that their actual behaviors need to catch up with their level of cognition and attitude toward it. This is 
in line with the results reported by Andersen et al. (2018), Avila et al. (2019), and Hesselink (2000), which also 
emphasized that students’ behavior of sustainable development is not sufficient in their daily life. Moreover, 
Avila et al. (2019) confirmed the lack of applicability and continuity of actions as a major barrier to practicing 
sustainable development. If the university's policy efforts to foster sustainable development remain one-off, 
it becomes difficult to expect sustainable development’s ultimate change and growth. 

In sequence, this study compared the level of sustainable development by gender and educational experi- 
ences and found their significant differences. This aligns with the findings conducted by Atik et al. (2022), Torb- 
jOrnssona (2011), Yoo and Oh (2016) and Zhou (2017). Yoo and Oh (2016) insisted that the difference between 
males and females may be somewhat limited, but gender with high sensitivity to sustainable development 
can be linked to behavior. In particular, this study found that female students had significantly higher levels of 
sustainable development cognition and attitude, which suggests that females are more aware and concerned 
about environmental issues than males. In this regard, Atik et al. (2022) confirmed that females preferred sustain- 
able lifestyles, such as purchasing eco-friendly products, and in this study, females’ sustainable development 
cognition was the highest at 3.93(shown in Table 2) compared to other areas of sustainable development. 

The current research found differences in sustainable development according to school or personal learn- 
ing experiences, corroborating the findings of prior studies (Gu, 2022; Hori & Fujii, 2021; Lee, 2019; Mahtab & 
Asghar, 2022). Schools can help college students understand the concepts and principles of sustainable de- 
velopment, and they can raise their cognition of sustainable development by understanding the meaning and 
background of sustainable development through classes (Mahtab & Asghar, 2022). Gu (2022) also argued that 
personal online learning opportunities linked to school classes contribute to further strengthening learning 
outcomes. In the process of actively collecting related digital information, sustainable development behavior 
can be further increased by positively affecting the values of college students. 


The Association of Cognition and Attitude toward Sustainable Development Behavior 


The current study revealed that both sustainable development cognition and attitude significantly impact 
behavior. Specifically, cognition acted as an influencing factor, and attitude acted as a mediating factor. These 
findings are consistent with previous studies conducted by Yoo and Lee (2014), Michalos et al. (2011), and Min 
(2019), which also found a significant correlation between cognition-attitude behavior and attitude-behavior 
of sustainable development. Yoo and Lee (2014) revealed that attitudinal factors mediate the association 
between environmental cognition and behavior, suggesting that attitude plays a crucial role in translating 
environmental knowledge into sustainable behavior. 

This study adds to the body of literature by providing empirical evidence for the structural causal associa- 
tion between cognition, attitude, and behavior toward sustainable development. In this regard, as Blake (2001), 
and Ma and Zhao (2008), sustainable development behavior affects complex factors by operating comprehen- 
sively. This study particularly revealed the full mediating effect of sustainable development attitudes. Moreover, 
it expands on previous research by providing empirical evidence of the mediating effect of attitude on the 
association between cognition and behavior. Applying these findings to the cognitive behavior model (Ellis, 
1978; Frijda, 1993), when students encounter sustainable development knowledge as an event, they perceive 
it through their own system or evaluate the knowledge in light of their attitudes. As a result, emotional or 
behavioral consequences are expressed. At this time, college students’ original attitude and values may cause 
an additional outcome that further strengthens the related sustainable development attitude. This finding has 
significant implications for sustainable development interventions, as it suggests that phased interventions 
targeting cognition and attitude may be more effective in promoting sustainable behavior. 
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The Moderation Effect of Informal Online Learning 


In this study, only the moderation effect of informal online learning was significant among the assumed 
personal characteristics within the relationship between sustainable development cognition-attitude-behavior. 
On the other hand, the study found no significant difference between groups according to the educational 
experience at the college. These findings are in line with prior study findings by Huang (2014), Yang et al. 
(2017), and Yi and Lu (2022). Huang (2014) and Yang et al. (2017) criticized the limitations of the current status 
of superficial sustainable development education in Chinese colleges, and this study is related to the assump- 
tion of sustainable development behavior as a dependent variable. In other words, the current college educa- 
tion is insufficient to induce changes in students’ sustainable development behavior. In addition, as Yi and Lu 
(2022) argued, related implementation strategies and preparation of alternatives are required due to the lack 
of research dealing with college students’ sustainable development behavior. 

In the difference test, female students’ attitudes and behavior toward sustainable development were 
higher than male counterparts. However, the causal relationship between these three factors showed that the 
difference according to gender was insignificant. These comparative research results are related to the studies 
of Huang (2014) and Min (2019). Rather than a tendency for female students’ high-level sustainable develop- 
ment attitudes to be consistently linked to behavior, personal characteristics by gender, such as ‘males are 
more actively interested in eco-technologies and sustainable energy (Huang, 2014)’ It can be interpreted that 
tastes are expressed more preferentially. In particular, since this study focused on students in major science, 
as Yu and Yang (2019) argued, the characteristics of each major, in addition to gender, may have an impact. 

In sum, these findings suggest the prospect of using informal online learning to foster sustainable devel- 
opment behavior and provide support for using informal online learning. Hori and Fujii (2021) and Kim and 
Yi (2012) confirmed the increase in personal learning using ICT among young people in the 21st century and 
identified information sharing and communication in the online space as the best tools for learning. Consid- 
ering the preferences and characteristics of college students in the MZ generation, we should recognize the 
significance of utilizing the digital space and ICT in education. Therefore, it is necessary to reflect on whether the 
national and policy attempts to foster sustainable development are being promoted in practice and whether 
the characteristics of university students as learners are actively reflected (Zhang, 2020). 


Conclusions and Implications 


This study aimed to examine the level of cognition, attitude, and behavior towards sustainable develop- 
ment and to verify the association among them through the moderation effect of informal online learning to 
enhance sustainable development behavior. In addition, various statistical processes were calculated to predict 
the final influencing factors for the accuracy of the research results. Therefore, the significant mediating role of 
sustainable development attitude and the moderation effect of informal online learning was confirmed. The 
study is expected to contribute to the sustainable development agenda by highlighting the need for targeted 
educational strategies to promote sustainable practices among young people. The above conclusions provide 
important messages for universities seeking to promote sustainable development education among college 
students. 

The study emphasizes the importance of prioritizing attitude improvement as a prerequisite for effective 
sustainable development practice. It is not enough to provide information and knowledge about the serious- 
ness of environmental issues. Sustainable development education must also aim to increase students’ sense 
of need and empathetic attitude towards sustainable development. Moreover, the study highlights the need 
for college students to take an active and positive approach to sustainable development education. Equipping 
students with the knowledge, skills, and attitudes needed to address the complex challenges facing the world 
today is crucial for promoting a sustainable future. To that end, universities should play a critical role in promot- 
ing sustainable development education among college students and prioritize this important agenda in their 
curricula and pedagogical practices. 

Universities also should develop and support a sustainable development education program that consid- 
ers students’ evolving needs and growth. In particular, universities should explore approaches that promote 
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behavioral change through personal life-oriented learning courses. This study proposes an experienced ap- 
proach to sustainable development education, which interconnects non-formal education and informal learn- 
ing in addition to university education. This holistic approach recognizes the interconnectedness of various 
personal experiences and suggests that sustainable development education should address these challenges 
in a comprehensive and integrated way. 


Limitations 


Nevertheless, this study has several limitations. First, since only college student participants were sur- 
veyed, the results may not be directly used by other students or groups with different cultural backgrounds. 
Second, the effects of cognition and attitude on sustainable development behavior have been considered, 
but the impacts of other potential variables may still exist. Therefore, to more comprehensively and practically 
understand sustainable development behavior in future studies, it may be considered to introduce factors 
such as family and social factors. 
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Abstract: Currently, nations grapple with 
a noticeable shortage of STEM talent, 
particularly within fields like agroforestry. 
The study applied the structural equation 
modelling and the mediation effect model 
to analyze the mechanisms through 
which STEM value perception influences 
STEM career preferences, as well as the 
roles played by STEM self-efficacy and 
STEM learning interest among 1,604 
undergraduates majoring in agroforestry. 
The results revealed that students’ STEM 
value perception was found to directly 
and positively correlate with their STEM 
career preferences. STEM value perception 
also has an indirect impact on STEM 
career preferences by influencing STEM 
learning interest and STEM self-efficacy. 
Furthermore, STEM value perception plays 
a sequential mediation role in promoting 
STEM learning interest, STEM self-efficacy 
and STEM career preferences. Based on 
these findings, it is recommended that 
institutions bolster STEM career education 
for students majoring in agroforestry from 
the outset of their enrollment, enhancing 
students’ professional awareness and 
identity, by addressing gaps in students’ 
value perceptions of STEM education. 
Furthermore, efforts to improve students’ 
interest in relevant courses and their 
confidence in learning these subjects can 
be beneficial. Ultimately, such initiatives 
can contribute to a more balanced 
development of students’ STEM career 
preference. 
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Introduction 


In order to adapt to the changes in the development of the world’s 
economy, society, science and technology and the profound changes in the 
demand for talent training, countries around the world have strategically for- 
mulated policies to promote the training of STEM talents in order to develop 
the national economy and increase the pool of scientific and technological 
innovation talents (NRC, 2011; OECD, 2012). However, despite the fact that 
governments have introduced policies and implemented reforms to improve 
the quality and quantity of STEM-related talent training at universities, few 
people are engaged in STEM-related careers after university graduation, 
and there is still a serious shortage of STEM talent (Gago et al., 2005; OECD, 
2018; Stipanovic & Woo, 2017). Consequently, numerous researchers have 
embarked on exploring the STEM career aspirations of students across all 
age groups, aiming to gain insights into their preferences within the realm 
of STEM careers (Carpi et al., 2017; Chachashvili-Bolotin et al., 2016; Lent et 
al., 2010; Zhan, et a., 2023). 

According to Social Cognitive Career Theory (SCCT), an individual’s 
career interest is determined by the core variables of self-efficacy, learning 
interest, outcome expectations, and learning experience, along the lines 
of Individual-Behavior-Environment (Lent et al., 2010). Currently, many re- 
searchers have explored the status of students’ STEM career preferences and 
the factors that influence students’ expectations of pursuing STEM-related 
careers, such as self-efficacy (Blotnicky et al., 2018), learning interest (Luo et 
al., 2021), impressions of STEM (Chen et al., 2022), family (Balt et al., 2023), 
formal or informal learning environments (Cheng et al., 2021; Drymiotou et 
al., 2021), and so on. While undeniably valuable, a majority of these studies 
have primarily concentrated on the elementary and secondary education 
tiers, leaving a noticeable gap in terms of studies conducted at the university 
level. The significance of this omission becomes particularly pronounced 
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when considering that university-level students, typically aged 18 to 23, have already made their major selections. 
This phase of education is especially pivotal for students within science and engineering fields, as it represents a 
crucial juncture for assimilating specialized knowledge, refining professionalism, cultivating a sense of professional 
identity, and nurturing a heightened professional interest (Liu et al., 2022). The STEM career inclinations exhibited 
by this cohort of students hold a stronger predictive power for their subsequent career trajectories within the field, 
surpassing the predictive significance observed in students of different age brackets (Liu et al., 2022). Consequently, 
delving into the STEM career propensities of students enrolled in science and technology programs at colleges 
and universities holds immense value. 

Amidst the swift progression of the latest global scientific and technological revolution, coupled with indus- 
trial transformations, the significance of agricultural and forestry disciplines has become even more pronounced. 
Pioneering technologies encompassing biotechnology, engineering advancements, information technology, and 
other forefront innovations are consistently permeating the realm of agriculture, fostering an innovative resurgence 
in agricultural science and technology. This phenomenon presents a series of intricate challenges and requisites 
for structuring disciplines and specializations, along with nurturing talent, within educational institutions focus- 
ing on agriculture (Ameyaw et al., 2017; Sharik et al., 2015). However, data showed that among all STEM majors, 
the proportion of students majoring in agriculture and forestry who are engaged in majors-related occupations 
after graduation is the smallest (Kim et al., 2022, World Bank, 2019). Parents and students generally perceive ag- 
riculture and forestry majors to be difficult to employ and of little value (Cosby et al., 2022). Therefore, studying 
the influence of this group’s STEM value perception on their STEM career tendency can help relevant professional 
teachers, schools and the state to analyze the academic situation, improve career education in agricultural and 
forestry colleges and universities, and cultivate urgently needed and scarce reserve agricultural and forestry sci- 
ence and technology talents. 

Based on the above background, this study explored the influence of STEM value perceptions of Chinese 
agricultural and forestry university students on their future STEM career preferences by means of a questionnaire 
survey, and explored the mediating roles of STEM learning interests and STEM self-efficacy, with a view to provid- 
ing references to improve the future STEM career preferences of this group of students. 


Literature Review 
STEM Career Preference 


Career preference encompasses an individual's future career intentions, along with their ambitions and an- 
ticipations for a specific occupational trajectory before entering the human resource market (Gottfredson, 1981). 
This personalized selection plays a pivotal role in shaping one’s prospective career orientation, carrying significant 
motivational weight (Davey & Stoppard, 1993). Within this context, STEM careers encompass domains such as 
engineering, healthcare, information technology, and more (Blotnicky et al., 2018). Synthesizing the concepts of 
STEM careers and career aspirations, STEM career preference can be defined as an individual's inclination or sub- 
jective readiness to pursue a future in STEM careers at a specific juncture (Mau & Li, 2018). The career preferences 
of students serve as a predictive and analytical lens into their future professional paths (Salehjee & Watts, 2015). 
Consequently, an array of studies has explored and scrutinized the current landscape of STEM career aspirations 
across students of varying ages, with a focus on three primary spheres. The first domain centers on investigating 
the present landscape of students’ inclinations towards STEM careers. For instance, Koyunlu Unlii and Dékme 
(2020) gauged the level of interest among Turkish secondary school students in STEM careers, employing the 
STEM Career Interest Survey (STEM-CIS) devised by Kier et al. (2014). The second realm delves into the principal 
factors influencing students’ attraction to STEM careers. Cheng et al. (2021), for instance, integrated 3D printing 
into STEM learning and examined its influence on students’ inclinations towards STEM careers. Similarly, Luo et al. 
(2021) probed the impact of stereotypes surrounding STEM careers on upper primary school students, highlighting 
the mediating roles of self-efficacy and outcome expectations. Moreover, So et al. (2020) explored the potential 
effects of elementary school students’ perceptions of STEM careers on their interest in the field. The third domain 
revolves around the implementation of specific methods aimed at fostering students’ fascination with STEM ca- 
reers. Chen (2022) and others adopted an approach involving hands-on STEM activities to heighten interest among 
elementary students in STEM careers. 
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STEM Self-Efficacy, STEM Value Perception, STEM Learning Interest, and STEM Career Preference 


Students’ value perception of a career refers to the individual’s expectation of the likelihood of success in 
developing a career with this career as well as the values that are identified with the career (Compeau, 2016). An 
individual’s motivation to pursue a particular career is determined by his value perception of that career (Salehjee 
& Watts, 2015). The easier an individual perceives himself to be valued in this occupation, the more likely he/she is 
to be motivated and thus the more motivated he/she is to pursue that occupation (Nugent et al., 2015; Zhang & 
Barnett, 2015). Prior research has found that students’ perceptions of value in STEM careers can had an impact on 
students’ STEM career expectations (Blotnicky et al., 2018; Heilbronner, 2009; Luo et al., 2021). However, despite 
the relationship between the two, few studies have explored how students’ perceived STEM values influence their 
career preferences towards STEM. 

Learning interest is a fundamental motivator and driver of learning (Dewey, 1913; Hidi & Renninger, 2006). In 
this study, STEM learning interest refers to students’ interest in learning STEM-related courses. Students’ interest 
in learning a subject is a key predictor of future participation in related careers (NRC, 2007). Currently, there are 
many studies that have examined the influence of students’ interest in STEM learning toward career preference, 
confirming that students’ interest in STEM learning is a significant positive predictor of students’ occupational 
dispositions (DeBacker & Nelson, 1999), and that students’ learning interest is a positive predictor of students’ 
self-efficacy (Jiang et al., 2022). In addition, self-efficacy has been also shown to play an important mediating role 
in STEM education (Cabell, 2021, Halim et al., 2021). 


Research Questions and Hypotheses 


In order to clarify how the STEM value perception influence the STEM career preference of university students 
majoring in agriculture and forestry, and then to better enhance students’ STEM career preference, this study posed 
the following research question: 


1. How does the perception of STEM value (VP) among undergraduate students majoring in agroforestry 
impact their preferences for STEM careers (CP)? 

2. What underlying mechanisms drive this influence, and how do STEM learning interest (LI) and STEM 
self-efficacy (SE) contribute to the pathway through which STEM value perceptions (VP) shape STEM 
career preferences (CP)? 


To answer the research question, below hypotheses were made about the influence mechanisms between 
the independent and dependent variables based on the existing literature. 


H1: Agroforestry students’ STEM value perceptions (VP) can directly influence students’ STEM career prefer- 
ences (CP); 

H2: Agroforestry students’ STEM value perceptions (VP) can influence STEM career preferences (CP) by 
influencing STEM learning interest (LI); 

H3: Agroforestry students’ STEM value perceptions (VP) can influence STEM career preferences (CP) by 
influencing STEM self-efficacy (SE); 

H4: There is a chain effect between STEM value perceptions (VP), STEM learning interest (LI), STEM self- 
efficacy (SE), and STEM career preferences (CP) of agroforestry students. 


The hypothetical model for this study is shown in Figure 1. 
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Figure 1 
Hypothetical Model 


Research Methodology 


Background 


This research employed a quantitative research methodology, specifically utilizing Structural Equation Model- 
ling (SEM), to explore the influences among STEM value perceptions, STEM learning interest, STEM self-efficacy, and 
STEM career preferences within the context of students majoring in Agriculture and Forestry. The study employed 
a standardized questionnaire to collect data. Prior to the main survey, a preliminary test was conducted with a 
subset of participants, which facilitated the refinement of the questionnaire through Confirmatory Factor Analysis 
(CFA) and reliability analysis. The final survey took place in March 2023, and an electronic distribution method was 
employed to administer the questionnaire to the entirety of students enrolled in agricultural and forestry programs 
at Zhejiang Agriculture and Forestry University, China. The coordination of this process was overseen by class 
lecturers, and the data collection spanned approximately one month. Subsequent to collecting the completed 
questionnaire, a rigorous analysis was undertaken employing statistical tools such as SPSS and Amos. The primary 
goal of this analysis was to uncover the extent to which students’ perceived values in STEM education influence 
their preferences for STEM careers. Additionally, the study sought to elucidate the underlying mechanisms that 
mediate this connection and rigorously test the research hypotheses. 


Participants 


To ensure a balanced representation, accuracy, and ease of sample collection, a purposive sampling method 
was employed to survey students from agriculture and forestry universities. Zhejiang Agriculture and Forestry 
University, dedicated to education and research in these fields, offers a diverse range of programs encompassing 
agricultural and forestry sciences. Furthermore, the university attracts students from various backgrounds and 
regions, which also guarantees the collection of a comprehensive and expansive sample. 

The formal questionnaire encompassed students from diverse academic years (freshmen to seniors), with ages 
ranging from 17 to 23 and an average age of 19. The distribution of the online questionnaire facilitated tracking 
of the respondents’ count, culminating in a total count of 1610 responses. The questionnaire was disseminated 
under the supervision of lecturers, and the electronic format included mandatory questions that needed to be 
answered comprehensively before submission. This meticulous approach led to a higher level of diligence and 
completion, consequently resulting in a minimal number of invalid responses. Finally, 1,604 (99.6%) valid question- 
naires were identified. Table 1 presents the demographic characteristics of the participants involved in the data 
collection process. 
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Variables Participants Percentage (%) 
Male 577 36 
ali Female 1027 64 
Agronomy 113 7 
Plant Protection 76 47 
Smart agriculture 18 11 
Forestry 170 10.6 
Biotechnology 75 4.7 
Ecology 70 4.4 
Agricultural Resources and Environment 178 11.1 
Horticulture 140 8.7 
Major Landscape Architecture 209 13 
Agricultural and Forestry Economic Management 115 7.2 
Animal Science 31 1.9 
Veterinary medicine 58 3.6 
Facility Agricultural Science and Engineering 86 5.4 
Tea science 36 2.2 
Cultural Industry Management (Tea Culture) 54 3.4 
Traditional Chinese Medicine 94 5.9 
Biopharmaceuticals 81 5 
Freshman 615 38.3 
fae Sophomore 490 30.5 
Junior 288 18.0 
Senior 211 13.2 


Instrument 


This study formed a questionnaire on students’ STEM learning interest, STEM self-efficacy, STEM values per- 
ceptions, and STEM career preferences consisting of four sub questionnaires. The instrument contains a total of 14 
items with a 5-point Likert scale (1=strongly agree, 2=agree, 3=neutral, 4=disagree, 5=totally disagree). The specific 


distribution of the questionnaire items is shown in Table 2. 


Table 2 
Distribution of the Questionnaire Items 


Sub-questionnaire 


STEM Learning Interest 
STEM Self-Efficacy 
STEM Value Perception 


STEM Career Preference 


Item 
1,2, 3,4 
5,6, 7,8 

9, 10, 11, 12 

13, 14 
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The STEM Learning Interest sub-questionnaire was structured into four distinct dimensions, mirroring interest 
in knowledge comprehension, engagement in activity participation, focus on skill acquisition and enthusiasm for 
course learning. Each dimension was represented by a singular item, inspired by the work of McGuire et al. (2021). 
Regarding the STEM Self-Efficacy sub-questionnaire, following the questionnaire’s development and validation 
by Luo et al. (2021), students’ STEM self-efficacy was categorized into four facets: understanding of knowledge, 
mastery of skills, aptitude for problem solving, and performance in tests. Each of these sections corresponded to 
an individual item. As for students’ perceptions of STEM value, guided by the research of Halim et al. (2018), it was 
compartmentalized into four dimensions: recognition within society, economic standing, prospects for employ- 
ment, and personal ability enhancement. A set of questions was designated for each dimension to capture students’ 
perspectives. To gauge students’ career inclinations, the study drew inspiration from Karen et al-s (2018) research. 
Specifically, two questions were employed: “In my future career, | will utilize knowledge linked to mathematics, 
science, engineering, and technology to resolve challenges” and “I possess a keen interest in pursuing a career asa 
scientist, engineer, or technologist.’These questions provided insights into students’ preferences for STEM-related 
career paths. 

In this study, the main factors were measured in the form of sub-questionnaires, so testing the quality of the 
measured data is an important prerequisite to ensure that the subsequent analysis is meaningful. First, the internal 
consistency of the scales was tested by the Cronbach reliability coefficient. As shown in Table 2, the reliability coef- 
ficient values of the sub-questionnaires are all above 0.8, and the instrument has good reliability. In addition, the 
Confirmatory Factor Analysis (CFA) model fit sub-questionnaire of STEM learning interest, STEM self-efficacy and 
the STEM career preferences was also required. The CFA was used to judge the model fit mainly by referring to ¥7/ 
df, RMR, RMSEA, CFI and other indicators. Among them, the cardinality freedom ratio is always greatly influenced 
by the sample size, therefore, in large samples, it is usually combined with other indicators for comprehensive 
evaluation. The results of model fitness tests for the subscale of STEM learning interest, STEM self-efficacy and the 
STEM career preferences are shown in Table 4. It can be seen that each indicator basically meets the requirements, 
and the model has an acceptable fitness. 


Table 3 
Reliability of the Sub-questionnaires 


Sub-questionnaire Cronbach’s a Items 
LI 928 4 
SE 948 4 
vP 884 4 
CP 837 2 
Total 948 16 
Table 4 
Model Fit 
Sub- 
questionnaire GF AGFI RMSEA NFI RFI IFI CFI RMR ald 
Ll .999 .993 026 .999 .998 1.000 1.000 .004 2.072 
SE .995 974 069 .997 .992 .998 998 006 7.662 
VP 1.000 .999 .001 1.000 1.000 1.000 1.000 001 195 
Reference Value >9 >9 <1 >9 >9 >9 >9 <1 <10 
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Based on the goodness of fit, convergent validity (Average Variance Extracted, AVE) and composite reliability 
(CR) were further examined. The standardized factor loadings of each item in the corresponding dimension were 
calculated by the established CFA model. The AVE and CR values were calculated. The CR > 0.8 and the AVE > 0.5 


(Table 5), which proves that the convergent validity and combined reliability are good. 


Table 5 
AVE and CR Results 

Path p AVE CR 

Lit <- LI 854 

LI2 <-- ul 894 
164 928 

LI3 <-- ul .890 

LI4 <-- ul 858 

SE1 <- SE 865 

SE2 <- SE 918 
821 948 

SE3 <- SE 927 

SE4 <- SE 912 

VP 1 <- vP 823 

VP2 < vP 790 
.661 886 

VP3 <-- VP 865 

VP4 < vP 170 

CP1 <-- CP 863 
122 838 

CP2 <- CP 836 


Ultimately, as depicted in the findings presented in Table 6, the discriminant validity assessment highlights 
that the standardized correlation coefficient between each pair of sub-scales remains below the square root of 
the corresponding dimension’s value. This outcome underscores the robustness of discriminant validity among 
all sub-questionnaires, signifying that each sub-questionnaire effectively distinguishes itself from others within 
the construct. 


Table 6 


Discriminant Validity Test 


Subscale 


LI 
SE 
vP 
cP 


AVE square root 


ul 


164 
814 
581 
835 
874 


SE 


821 
537 
166 
.906 


Data Analysis 


vP 


661 
679 
813 


cP 


122 
.850 


Following the collection of data, an analysis was performed employing SPSS 27.0 and Amos 27.0. The initial 
steps encompassed reliability assessments, validity analyses for confirmation, evaluations of convergent validity, 
combined reliability measurements, and differential validity tests. These evaluations aimed to ascertain that the 
indicators adhered to the prescribed model fit standards. The research subsequently proceeded to examine the 
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interconnections among students’ STEM learning interest, STEM self-efficacy, STEM value perceptions, and STEM 
career preferences. In this analysis, STEM value perceptions served as the independent variables, while STEM 
learning interest and STEM self-efficacy functioned as mediating variables. The primary focus was on STEM career 
preference as the dependent variable. This examination encompassed the calculation of both the direct impact of 
STEM value perceptions on STEM career expectations and their indirect influence via STEM learning interest and 
STEM self-efficacy pathways. 


Research Results 
Descriptive Statistics and Normality Test 


Descriptive statistical analysis and normality tests were conducted on the overall student performance, and 
the results are shown in Table 7. According to the results of the overall descriptive statistical analysis, it can be 
seen that the mean score of each Subscale is between 3 and 4. This shows that college students in agriculture and 
forestry as a whole had above-average levels of STEM learning interest, STEM self-efficacy, STEM value perception, 
and STEM career preferences. The highest mean value of the four elements is the STEM value perception, which 
indicates that students all believed that knowledge related to STEM subjects had some value. Also, the lowest is 
STEM self-efficacy, indicating that students’ confidence in their ability to master learning and activities related to 
STEM subjects needs to be further improved. 

In addition, for each item, a normality test was conducted using kurtosis and skewness. The kurtosis coef- 
ficients of the data in this study are within 3 in absolute value and the skewness coefficients are within 8 (Table 7), 
indicating that the measured items met the approximate normal distribution. 


Table 7 
Descriptive Statistics and Normality Test 


Subscale Item M SE Skewness Kurtosis M SD 

LI 3.08 1.006 - 290 - 068 
LI2 3.05 998 - 242 - 194 

ul 3.155 3.610 
LI3 3.23 993 - 441 - 028 
LI4 3.26 983 - 429 072 
SE1 3.13 .967 - 356 - 020 
SE2 3.09 974 - 310 - 002 

SE 3.123 3.584 
SE3 3.19 954 - 414 115 
SE4 3.08 .960 - 265 023 
VP1 3.56 912 - .776 872 
VP2 3.64 912 - 872 1.026 

VP 3.55 3.134 
VP3 3.51 898 - 633 156 
VP4 3.47 916 -.656 658 
CP1 3.21 1.008 - 360 - .038 

CP 3.18 1.911 
CP2 3.15 1.052 - .270 -.271 


Correlation of STEM Learning Interest, STEM Self-Efficacy, STEM Value Perception and STEM Career Preferences 


Pearson correlation analysis was conducted on the scores of four sub-questionnaires: students’ STEM value 
perception, STEM learning interest, STEM self-efficacy, and STEM career preferences. Exploratory analysis of the 
correlations among multiple variables shows that the correlation coefficients r between each variable are greater 
than 0 and all are significant (Table 8). This indicates that there was a significant positive correlation between stu- 
dents’ STEM value perception, students’ STEM learning interest, STEM self-efficacy, and STEM career preferences 
supporting further regression analysis and mediation effect tests. 
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Table 8 


Pearson's Correlation Coefficient Between Sub-questionnaires 


Ll SE vP cP 
LI 1 
SE .166** 1 
VP 041™* 502** 1 
cP 139* .683** .588** 1 


**_ Significant correlation at the .01 level (two-tailed). 


Path Hypothesis Testing 


The path hypothesis of the model was tested, and the data obtained are shown in the table. As can be seen 
from Table 9, the p-values are less than .001 for each path of the model, indicating that there was a significant 
positive predictive relationship between the independent variables on the dependent variable. That is, students’ 
interest in STEM learning, perceptions of STEM values, and STEM self-efficacy all significantly and positively pre- 
dicted students’ STEM career preferences. 


Table 9 
Direct Effects Test 
Path B S.E. C.R. p 
ul > VP 581 .033 21.147 — 
SE >> Ll 157 026 29.703 sa 
SE > VP .097 .028 4.335 ne 
CP >> Ll 511 .038 14.034 = 
CP >> SE .204 .033 6.104 ase 
CP >> VP 273 029 11.458 iia 


Based on the results of the hypothesis test, the path is derived as shown in Figure 2. 


Figure 2 
Path Testing 


OS81F** 0.204*** 


0.0978" “O511F** 
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The total influence of students’ STEM value perceptions on students’ STEM career preferences was tested 
with a confidence interval of [0.3345, 0.3828] without 0 and an effect size of 0.3587, indicating that students’ value 
perceptions of STEM had a significant predictive effect on STEM career preferences (Table 10). After that, the me- 
diating variables of students’ STEM learning interest and STEM self-efficacy were added for testing, at which point 
the confidence interval of students’ perceptions of STEM values on students’ STEM career preferences is [0.0114, 
0.1224], which do not contain 0, indicating a significant direct effect. The confidence interval for the total indirect 
effect is [0.0126, 0.1895], which does not contain 0, indicating that the total indirect effect was significant, and the 
mediating effect existed and was partially mediated. The amount of total indirect effect accounts for 59.63% of 
the total effect. The indirect effects were divided into three paths, none of which confidence intervals contain 0, 
indicating that the mediating effect of STEM learning interest, the mediating effect of STEM self-efficacy, and the 
chain mediating effect of STEM learning interest and STEM self-efficacy all held true. 


Table 10 
Bootstrap Analysis of Mediation Effects 


Bootstrap 95% Cl 
Path B S.E. p 
Lower Upper 
Total Effect .3587 .0123 13345 3828 Significant 
Direct effect 1447 0114 1224 .1670 Significant 
Total Indirect Effect 2139 0126 1895 2383 Significant 
Path 1: VP —» Ll —» CP 1428 0121 1196 1169 Significant 
Path 2: VP —» Ll —» SE —» CP .0176 .0042 .0103 .0264 Significant 
Path 3: VP —» SE —» CP .0536 .0077 0391 .0695 Significant 


Discussion 


This study examined the influence of STEM value perceptions of university students majoring in agriculture 
and forestry on their STEM self-efficacy, STEM learning interest and STEM career preference. A sample of 1604 
agriculture and forestry students from the first to fourth year of university was chosen to be analyzed with their 
STEM value perception as the independent variable, STEM career preference as the dependent variable, and STEM 
learning interests and STEM self-efficacy as mediating variables. 

The results underscore that students’ STEM value perceptions possess a direct influence over their prefer- 
ences for STEM careers. Previous research has revealed that heightened STEM career knowledge correlates with 
an increased likelihood of selecting STEM professions among secondary school students (Blotnicky et al., 2018; 
Drymiotou et al., 2021). Additionally, STEM stereotypes can predict the interest secondary school students show in 
STEM careers (Shen et al., 2014), while there are noteworthy implications on primary school students’ perceptions 
and interest in STEM careers (So et al., 2020). This study extends these previous findings by delving deeper into 
the domain of college students majoring in agriculture and forestry. It specifically shows how the perceived value 
of STEM-related knowledge influences the likelihood of such students opting for STEM careers. This research rein- 
forces the significance of STEM value perceptions in shaping career aspirations within this student demographic. 
Given the historical trend of agriculture and forestry majors facing lower popularity, coupled with the prevalent 
belief that career prospects in these fields are limited, it becomes pivotal to educate these students about their 
career options upon enrollment, clarifying the value and potential trajectories that their chosen major can offer 
(Drymiotou et al., 2021). 

Furthermore, this study demonstrates that STEM value perceptions can influence students’ pursuit of STEM 
careers through their impact on STEM learning interest and STEM self-efficacy. This correlation aligns with prior 
research indicating that students’ interest in learning and their perceived self-efficacy significantly impact their 
expectations for STEM careers (Brown et al., 2016; Lent et al., 2018; Potvin & Hasni, 2014). It is also consistent with 
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findings showing that students’ perceptions of STEM values can indirectly influence their self-efficacy and, in turn, 
their career expectations (Garriott et al., 2017; Luo et al., 2021). Building upon this foundation, the present study 
not only validates the role of STEM value perceptions in shaping STEM learning interest and self-efficacy but also 
unveils a chain mediating effect encompassing STEM value perceptions, STEM learning interest, STEM self-efficacy, 
and STEM career preferences. 

The identified chain effect highlights that when students perceive a specific piece of knowledge as valuable, 
their interest in learning it amplifies, fostering greater enthusiasm and self-efficacy for mastering the subject. Con- 
sequently, this heightened self-efficacy drives their career expectations linked to that particular field of knowledge 
(Cromley et al., 2016; Hidi & Renninger, 2006; Liu et al., 2022). As a result, educators hold the potential to enhance 
students’ inclinations toward STEM careers by focusing on elevating their perceptions of the value of STEM. This, 
in turn, can stimulate greater interest in STEM-related courses and bolster students’ confidence in their ability to 
excel in these areas. 


Conclusions and Implications 


This study aimed to explore the influence of STEM values perceptions on the STEM career preferences of 
undergraduates majoring in agriculture and forestry. Through SEM analysis and mediation effect analysis, it has 
been found that students’ STEM values perceptions directly influence students’ STEM career preferences, as well 
as influence students’ career preferences by influencing their STEM learning interests and STEM self-efficacy, and 
there is also a chain-mediated effect between students’ STEM values perceptions, STEM learning interests, STEM 
self-efficacy and STEM career preferences. The results extended previous research that did not identify factors 
influencing STEM career dispositions among agroforestry students at college and university and highlighted the 
importance of professional perceptions in students’ future career choices, as well as the important mediating role 
of enhancing students’ interest in learning and self-efficacy. Therefore, a few suggestions are provided for teach- 
ers to cultivate students’ STEM career preferences in the future. Firstly, teachers should pay attention to creating 
informal learning environments, such as extracurricular activities, to stimulate students’ interest in STEM learning 
when conducting teaching. Second, teachers should pay attention to improving students’ knowledge transfer 
and application abilities, cultivating students’ problem-solving skills, and helping students to improve their sense 
of self-efficacy. In addition, in the future, educators should pay more attention to students’ perception of STEM 
values, actively carry out career education, and help agricultural and forestry majors to improve their perception 
of career values and their sense of professional and occupational identity. 

There are some limitations in this study, due to time and other limitations, there are many factors influencing 
students’ career preferences, this study only chose students’ STEM value perceptions, STEM learning interest and 
STEM self-efficacy for the study, and didn’t add the family background and other factors, in addition, in the spirit 
of the principle of convenience sampling, this study into the selection of a province of agriculture and forestry 
majors for the survey, in the future, the STEM career tendency of students majoring in agriculture and forestry in 
other provinces will be explored. 
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Appendix 


Survey for STEM Career Preference, Learning Interest, Self-Efficacy, 
and Value perception of Agroforestry Undergraduates 


Dear Student, 


Thank you for participating in this survey, which is designed to get a basic picture of your interest in STEM 
(Science, Technology, Engineering and Mathematics) related subjects and your views on STEM related careers. The 
questionnaire is anonymous responses and is for research use only, so you can feel free to fill it out, and | hope you 
will follow your heart’s true thoughts and fill it out carefully, thank you! 


LI1 Learning knowledge about Science, Technology, Engineering and Mathematics is interesting 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


LI2 | enjoy completing tasks related to science and engineering courses 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


LI3 The Science, Technology, Engineering and Mathematics related courses | take are relevant to my life 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


LI41 am curious about discoveries made during activities and events related to Science, Technology, Engineering 
and Mathematics 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


SE1 I think | am capable of solving problems encountered in my Science, Technology, Engineering and Mathemat- 
ics studies 


A totally disagree, B disagree, C neutral, D agree, F strongly agree 


SE2 |am confident that | can perform well in learning knowledge related to Science, Technology, Engineering and 
Mathematics 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


SE3 | believe | can acquire skills related to Science, Technology, Engineering and Mathematics courses and experi- 
ments 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


SE4 1am confident that | can do well in exams related to Science, Technology, Engineering and Mathematics courses 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


VP1 Professions in Science, Technology, Engineering and Mathematics enjoy a high level of social prestige 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


VP2 Occupations in Science, Technology, Engineering and Mathematics have higher earnings 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


VP3 Jobs in Science, Technology, Engineering and Mathematics have more employment opportunities 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


VP4 If | workin Science, Technology, Engineering and Mathematics jobs, | can improve my teamwork skills, problem 


solving skills, etc. 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 
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CP1 | will use knowledge of Science, Technology, Engineering and Mathematics to solve problems in my future career. 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 


CP2 | am interested in becoming a scientist, engineer or technologist 
A totally disagree, B disagree, C neutral, D agree, F strongly agree 
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DATA IN CHINA: EXPLORING 
STUDENTS’ PROBLEM-SOLVING 
STRATEGIES IN COMPUTER- 
BASED SCIENCE ASSESSMENT 
FEATURING INTERACTIVE TASKS 
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Chun-Yen Chang, 
Yueyang Shao, 
Zhi Liu, 

Hao Zhou, 

Jian Liu 


Introduction 


The rapid growth of science and technology has led to fundamental and 
momentous changes in the societies of the 21st century. These changes are 
reflected in the types and complexities of problems encountered in everyday 
life. To successfully address these problems, problem-solving competency 
that focuses on in-depth exploration, critical thinking, and analytical reason- 
ing in the process of understanding and applying knowledge is necessary and 
crucial for individuals (Lavoie, 1993; NRC, 1996). Problem-solving competency 
is an essential foundation for students to actively participate in social activities 
and engage in meaningful learning in the 21st century (NRC, 2010; OECD, 
2013). The ability to solve real-world problems is a crucial component of 
scientific literacy and a significant goal of science education (NRC, 1996; NRC, 
2012). Scientific problem-solving competency can help students construct 
a systematic knowledge structure and fully apply knowledge to cope with 
the challenges caused by the rapid development of science and technology 
(Friege & Lind, 2006; NRC, 1996). 

To better cultivate students’ ability to solve scientific problems, the 
processes of solving problems and students’ struggles to seek solutions 
need to be explored. Recently, computer-based assessments that support 
human-computer interaction have emerged in various problem-solving 
domains (Jiang et al., 2021; Vendlinski & Stevens, 2002). Computer-based 
assessments record detailed interactions between the problem solver and 
the task environment and thereby capture salient solution processes (Chung 
& Baker, 2003; Provasnik, 2021). The rich process data generated from the 
interactive items reflect the processes of answering the items and provide 
tremendous opportunities to reveal the cognitive and behavioral processes 
of the problem solvers. A growing number of studies have focused on process 
data to reflect students’ cognitive processes during problem solving. For 
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Abstract. Students’ problem-solving 
strategies and the differences among 
strategy groups were explored by analyzing 
the process data collected during student 
interactions with computer-based science 
items. Data were gathered from 1516 
eleventh-grade students from 4 schools 

in China. Analyses of the sequences of 
students’ response actions revealed that 
the students were divided into four strategy 
groups when designing experiments to 
solve scientific problems: the scientific and 
rigorous strategy (18.5%), scientific and 
less rigorous strategy (25.4%), incomplete 
strategy (31.5%), and chaotic strategy 
(24.6%). The heatmaps of response actions 
for each strategy and the frequencies of the 
most representative response sequences 
were further explored to understand the 
students’ detailed trajectories. The results 
showed that successful problem solvers 
were generally inclined to explore all 
possibilities of experimental combinations 
and design experiments scientifically and 
rigorously based on the relevant scientific 
principles. Moreover, the timestamps 

of response actions were explored to 

show that the students who adopted the 
scientific and rigorous strategy spent more 
time seeking solutions, suggesting that 
students may need sufficient time to solve 
complex and authentic scientific problems. 
The findings enrich the literature on using 
process data to address theoretical issues 
in educational assessment and provide 
students with individualized instructional 
needs for teachers to improve students’ 
scientific problem-solving competency. 
Keywords: process data, scientific 
problem-solving, computer-based 
assessment, China 
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example, some attempts have been made to explore the patterns of log files obtained from the National Assess- 
ment of Educational Progress (NAEP) and the Programme for International Student Assessment (PISA) (Bergner & 
von Davier, 2019; Greiff et al., 2015). 

However, it is still challenging to derive specific implications from the behavioral patterns found in log files 
because the cognitive process during problem solving is a black box and it is difficult to make sense of the hun- 
dreds of pieces of information students produce when solving problems in computer-based assessments (Greiff 
et al., 2015). Therefore, further exploration of process data is necessary to obtain more substantial findings, which 
can inform teachers and policymakers about students’ cognitive processes during problem solving. Meanwhile, 
limited research has been performed to emphasize the implementation of independently developed interactive 
assessments in China and the exploration of Chinese students’ cognitive processes of scientific problem solving. 
Hence, process data were utilized to explore cognitive strategies that eleventh-grade students adopt when solving 
interactive scientific tasks developed in China. 


Literature Review 
Scientific Problem-solving Competency 


Problem-solving competency determines how successful an individual will be at finding solutions to chal- 
lenges in life (Lazakidou & Retalis, 2010; OECD, 2013). Mayer proposed a widely accepted viewpoint that problem 
solving is the cognitive process of transforming a given situation into a target situation when there is no obvious 
solution (Mayer, 1992). The PISA 2012 problem-solving framework also stated that problem-solving competency 
is the ability of an individual to transform a given problem situation into a target situation by understanding the 
problem and solving it when the solution is not obvious and the individual's emotional willingness to participate 
in the problem solving (OECD, 2013). Hence, researchers emphasize that problem solvers explore the paths of 
solving problems from the initial state to the final goal state. Problem solving is a complicated process filled with 
continuous exploration from an initial state to a target state, which may involve multiple cognitive and emotional 
variables (OECD, 2013; Reid & Yang, 2002). 

Consistent with the aforementioned viewpoints, researchers have stated that problem solving in the field of 
science education is a process of scientific inquiry to some extent (Gayford, 1989). Students need to have some 
scientific knowledge and process skills to solve scientific problems successfully (Chang et al., 2007). The process 
of solving scientific problems involves the generation and explanation of ideas (Chang & Weng, 2002). Students 
need to think divergently to seek solutions to solve problems and need to explain and evaluate various solutions 
to choose the final one. Although researchers have considered scientific problem-solving competency from various 
perspectives, they are generally aware that scientific problem solving is a process of inquiry that requires a series 
of scientific process skills. Therefore, scientific problem-solving competency in this study involves using a series of 
scientific process skills combined with scientific knowledge to seek solutions and evaluate them to solve complex 
problems relevant to science and technology in daily life. 


Assessment Framework for Scientific Problem-solving Competency 


Researchers have stated that there is domain-general problem solving that pays attention to problem situ- 
ations across domains and domain-specific problem solving that focuses on specific educational areas, such as 
mathematics and science (Greiff & Neubert, 2014; Sternberg, 1995; Sugrue, 1995). Although the scientific problem- 
solving competency in this study is domain-specific problem solving, the studies of domain-general problem solving 
have also been instructive for us to consider our assessment framework because the two lines of research show 
considerable overlap. One of the most well-known assessments in the area of problem solving is the PISA 2012, 
which utilized a framework for domain-general problem solving (Greiff et al., 2014). This framework underlines 
a computer-based test design in which the students need to deal with daily real-life problems such as configur- 
ing an MP3 player. Students must explore and understand the problem situation (exploring and understanding), 
represent the problem by constructing representations, formulate hypotheses by identifying the relevant factors 
in the problem (representing and formulating), devise and carry out a plan to reach the goal state (planning and 
executing), monitor their progress and reflect on solutions from various perspectives (monitoring and reflecting) 
(OECD, 2013). Therefore, problem-solving competency involves formulating a hypothesis, devising and carrying 
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out a plan to solve the problem, and monitoring and critically evaluating problem-solving solutions, which are 
attributes of scientific inquiry. 

Some studies put problem solving in a special, domain-specific domain based on its context. Polya originally 
proposed a four-stage problem-solving model and revised it in mathematics education, including understand- 
ing the problem, devising a plan, trying and carrying out the plan, and monitoring and reflecting on the solution 
(Polya, 1957). The four-stage problem-solving model is consistent with the problem-solving framework in the PISA 
2012 to some extent. After Polya, some studies developed a problem-solving framework and assessment in the 
field of science. Some researchers have proposed that problem solving in science education has four overlapping 
and interactive aspects: problem posing, problem approaches, problem solutions, and communication (Mcin- 
tosh, 1995). Problem posing includes extracting a science problem from a realistic situation, making or revising a 
hypothesis, and planning experiments. Problem approaches involve controlling variables in an investigation, col- 
lecting appropriate data, and revising approaches when warranted by new evidence. Problem solutions include 
developing various solutions to the same problem, comparing the results, and making conclusions based on data. 
Communication focuses on communicating procedures, interpretations, and thinking pathways. Hence, the scien- 
tific problem-solving process has obvious scientific inquiry attributes and involves various scientific process skills 
(Eysenck & Keane, 2000; Gayford, 1989; Ross & Maynes, 1983). Moreover, scientific discovery as dual search (SDDS), 
an important model utilized to explain the process of scientific problem solving (Schauble, 2003), perceives scientific 
problem solving as a coordinated search in two problem spaces, the hypothesis space and experiment space (Klahr 
& Dunbar, 1988). In the hypothesis space, the initial state is domain knowledge, and the target state is hypotheses. 
Search in the experiment space is directed toward experiments that discriminate between rival hypotheses and 
yield interpretable outcomes (Eysenck & Keane, 2000). It is obvious that scientific problem solving is a process of 
scientific inquiry to some extent (Gayford, 1989). Based on what scientists do when designing experiments and 
solving scientific problems, problem solving involves various skills, including formulating hypotheses, designing 
experiments, recording information, and judging the data collected (Ross & Maynes, 1983). Researchers’ emphasis 
on experimental problem solving further indicates that the design of inquiry experiments is a crucial component 
of scientific problem-solving competency. 

Reasoning is another component of scientific problem-solving competency. Educational researchers in the 
earth and space sciences claim that problem-solving skills include domain-specific knowledge, reasoning skills, 
and attitudes (Chang et al., 2007). Researchers in chemistry state that reasoning ability is highly related to students’ 
ability to solve stoichiometric problems successfully (Robinson & Niaz, 1991). The PISA2012 problem-solving assess- 
ment framework also clearly puts forward the importance of reasoning, arguing that problem-solving processes 
rely on one or more reasoning skills (OECD, 2013). Additionally, explanations for problem solutions necessary for 
evaluating proposals are perceived as another component of scientific problem-solving competency. Studies 
suggest that problem solvers tend to articulate the steps to a problem solution and why they choose this solution 
(Chang & Weng, 2002; Chi et al., 1994). Successful problem solvers produce self-explanatory and principle-based 
explanations to effectively relate the solution to the principles or knowledge in science (Camacho & Good, 1989). 

Three components, including the design of scientific inquiry, scientific reasoning, and scientific explanation, 
are perceived as crucial dimensions in the assessment framework for scientific problem solving in this study. 


Use of Process Data to Reflect Students’ Cognitive Processes in Computer-based Assessments 


It is essential to understand the students’ cognitive processes in solving scientific problems and to identify 
students who are struggling with these processes for further instruction (Bergner & von Davier, 2019; Greiff et al., 
2015). With the conversion of science assessments from paper-and-pencil to computerized formats, process data 
derived from interactive item types emerged (Jiang et al., 2021; Provasnik, 2021). Log files are data sources for 
process data that reflect the cognitive and metacognitive processes involved in completing test tasks (Provasnik, 
2021). Process data provide fine-grained information about how students plan, select, and execute various problem- 
solving strategies to find a solution. Therefore, the crucial role of process data is to delineate how students solve 
problems, not just whether they do. Especially when process data are combined with theoretical frameworks and 
response data, abundant opportunities are afforded to study problem solvers’ paths to a solution (Provasnik, 2021). 

Process data mainly include detailed records of student interactions with the computer system, such as action 
sequences when solving simulation-based science tasks, answer change behaviors, and the timestamps of these 
actions (Bergner & von Davier, 2019). For computer-based science assessment focusing on interactive tasks, an 
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important piece of process data is the sequences of problem solvers’ response actions. These sequences provide 
insights into the strategies problem solvers frequently apply to solve scientific problems. For example, researchers 
exploited log files containing the process data of problem-solving behavior obtained from the 2012 cycle of the 
PISA (Greiff et al., 2015). Log-file analyses were conducted to explore whether the vary-one-thing-at-a-time (VO- 
TAT) strategy that students employed while solving the climate control item was related to their performance and 
then to identify several groups of students according to their exploration strategies. The classification of response 
strategies based on the analyses of problem solvers’ action sequences is a good way to identify classes of problem 
solvers and their specific needs (Arslan et al., 2020; Jiang et al., 2021). 

Another crucial piece of process data that researchers pay attention to is the timestamps of response ac- 
tions. Extensive studies have explored total response time on test questions as an indication of motivation (Lee & 
Jia, 2014; van der Linden, 2008). The timestamp data obtained from NAEP have been used to show the speed at 
which students compose written responses to test prompts and reflect various patterns and amounts of writing 
(Bergner & von Davier, 2019; Provasnik, 2021). Moreover, some research indicated that a long response execution 
time might suggest that the problem solvers are stumped by a problem; a short execution time might represent 
their high proficiency or rapid guessing (Guo et al., 2016; Lee & Jia, 2014). Other researchers claim that successful 
problem solvers devote considerable time to understanding the problem situation and seeking solutions (Gong et 
al.,2020). When constructing solutions to problems, experts generally devote sufficient time to effectively develop 
a strategy for creating a solution that is different from that of novices, who often approach problems haphazardly 
(Yerushalmi & Eylon, 2015). The aforementioned conclusions drawn on the analyses of response time are different 
and even opposite. This study can provide further information about the patterns of students’ cognitive processes 
when solving scientific problems. 


Problem Statement 


As mentioned earlier, due to the complexity of the scientific problem-solving process, there is much room 
for improvement in analyzing students’ cognitive strategies when solving scientific problems. By categorizing 
response strategies, categories of problem solvers and their specific needs can be identified, and this provides valu- 
able reference and evidence for educational intervention to improve problem solvers’ scientific problem-solving 
competency (Arslan et al., 2020; Jiang et al., 2021). However, there has been little research on Chinese students’ 
classification of cognitive strategies for solving scientific problems from the perspective of process data. Therefore, 
the purpose of the present study was to explore the classification of students’ experimental design strategies when 
solving scientific problems and the differences across strategy classes by analyzing process data. Accordingly, this 
study was conducted by analyzing Chinese students’ sequences of actions and response times with a focus on 
three research questions: 

RQ1: Whatis the classification of experimental design strategies that eleventh-grade Chinese students apply 

when solving scientific problems? 

RQ2: What is the detailed trajectory of each class of experimental design strategies? 

RQ3: Is there a significant difference in response time across strategy classes? 


Research Methodology 
Research Design 


The methodology employed in this study was that of a quantitative analysis. The computer-based science 
assessment was administered to student participants in July 2021, and quantitative data were obtained. Process 
data including the logs of students’ actions and the corresponding timestamps were analyzed. Descriptive statistics 
and Analysis of Variance (ANOVA) revealed the patterns of students’ strategies when solving scientific problems. 
First, descriptive statistics of students’ actions were performed to present a classification of experimental design 
strategies. Second, descriptive statistics were performed on students’ response sequences to reveal the detailed 
trajectories of each category of experimental design strategies. Finally, to explore the differences in response time 
across strategy classes, descriptive statistics and ANOVA tests on response time were conducted. 
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Participants 


In this study, a convenience sample of four high schools from a city (abbreviated as “M”) located in southwest 
China was recruited. The sample schools, including two urban and two suburban schools located in four regions 
in M city, provide ordinary educational resources to students. All four schools were in the middle rank (top 50.0%) 
of the education quality in M city. The students who attended the schools could be described as belonging to 
urban and suburban areas, with the school serving low, middle, and upper-income families. Students in grade 
eleven from the four sample schools participated in the test at the end of the spring term of the 2020-2021 aca- 
demic year, which constituted a total sample of 1516 students (57.0% female, 43.0% male). These students were 
between the ages of 16 and 17. The process data of students’ response actions to the inquiry task in subtask 1 of 
the Handy Freezer (abbreviated as “HF,’ see Figure 1) task needs to be collected to analyze their problem-solving 
strategies. This inquiry task requires students to continually click buttons and record the experimental data. Stu- 
dents’ response actions were recorded by the computer systems every 100 milliseconds as process data to show 
the real-time change in students’ responses. Eventually, the process data of 1516 students’ action sequences and 
response times were recorded for the analyses. 


Instrument 


Acomputer-based scientific problem-solving assessment featuring interactive tasks was developed. Comple- 
tion of tasks required only basic computer skills such as clicking on virtual buttons and sliders. The assessment 
instrument was developed based on the aforementioned framework of scientific problem-solving competency 
containing three elements (design of scientific inquiry, scientific reasoning, and scientific explanation). The instru- 
ment included three tasks consisting of several subtasks and covered the subjects of Physics, Chemistry and Biology. 
The Physics task consisted of five subtasks; the Chemistry task was made up of six subtasks; and the Biology task 
consisted of six subtasks. The programming tool JavaScript was utilized to form tasks, and the tasks were embedded 
in the test system OpenCT (https://open-ct.com). OpenCT is an interactive evaluation system based on Metaverse 
and educational big data for assessing higher-order thinking skills. The tasks were developed by a research team 
of three researchers (the first, third and fourth authors) and two technicians. Two of the three researchers were 
skilled in designing tasks associated with scientific literacy assessment. The other researcher was proficient in as- 
sessment theory. The two technicians were adept at designing interfaces for interactive tasks. One task was original 
and belonged to biology. The other task, belonging to physics, was adapted from an item developed by a research 
group from Guangxi Normal University. This item was included in a large-scale paper-and-pencil assessment to 
assess students’ scientific literacy in mainland China. Another task, belonging to chemistry, was retrieved from the 
work of a research team at the National Taiwan Normal University. The instrument in this study was revised accord- 
ing to the comments from three rounds of student interviews and two rounds of expert reviews. Each round of 
student interviews included eighteen eleventh-grade students; the first round of expert reviews contained three 
professors who were good at assessing scientific literacy; and the second round of expert interviews included four 
skilled professors in designing science tasks. The Cronbach's alpha for this instrument is .863. The comparative fit 
index (CFI), Tucker-Lewis index (TLI), and root mean square error of approximation (RMSEA) are key indicators of 
model fitness. The values for validity (CFI= .82, TLI= .80, RMSEA= .08) of this instrument show that the model is a 
moderate fit (Karadakal et al., 2015). 

Students had 60 minutes to complete the test. This study analyzed the process data collected from student 
interactions with one inquiry task in subtask 1 of the HF task (see Figure 1). The Chinese version of this inquiry 
task was translated into English for reading convenience. The inquiry task involved assessing the ability to design 
scientific inquiry included in the framework of scientific problem-solving competency and evaluating students’ 
knowledge on the scientific content area of evaporation in physics. On this task, students were instructed to click 
the buttons representing the “Water Level” and “Towel Position” to explore the condition for the best refrigeration 
effect. To solve this task, they needed to (1) click the “Water Level” button, (2) click the “Towel Position” button, (3) 
click the “Record” button to record the experimental data for the selected combination of “Water Level” and “Towel 
Position,’ and (4) continuously explore in this way and draw a preliminary conclusion about the condition for the 
best refrigeration effect according to the experimental data. Detailed logs of the actions and the corresponding 
timestamps were recorded in-process data and used for analysis. 
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Figure 1 


Subtask 1 of a Task Named Handy Freezer 


(Vv) Instauction 1 (Oy InsTRUCTION 2 C3} Sustask 1 Sustask 2 SUBTASK 3 Sustasx 4 


Towet Position Water Lever TEMPERATURE IN BOWL 


Towel Posiion 1 Water LeveL 1 30.0°C 


Measures 


Response actions. The sequences of actions on the inquiry task in subtask 1 were used to classify the problem- 
solving strategies according to the principles of designing experiments. The overall numbers of each experimental 
combination within each strategy group and the frequencies of the most representative response sequences were 
analyzed to explore the concrete trajectory of each strategy group. In this study, clicking the button representing 
the“Towel Position” and “Water Level” and recording the experimental data was conceived as a set of response ac- 
tions recorded as log data. There were three towel positions and five water levels (see Table 1). Specifically, “Towel 
Position 1” meant the towel covered the lid of the bowl, and “Towel Position 2” meant the towel was long enough 
to touch the bottom of the bowl. There were fifteen combinations of variables in total (see Table 2). The combina- 
tion of “Towel Position 2” and “Water Level 2” (code [2, 2]) was the condition for the best refrigeration effect. We 
classified the problem solvers’ response actions based on the principles of designing controlled experiments to 
better understand students’ problem-solving strategies. 


Table 1 
Variables and Corresponding Codes 


Variables Specific Circumstance Corresponding Codes 
No Towel 0 
Towel Position Towel Position 1 1 
Towel Position 2 2 
Water Level 0 0 
Water Level 1 1 
Water Level Water Level 2 2 
Water Level 3 3 
Water Level 4 4 
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Table 2 


Experimental Combinations and Corresponding Codes 


Experimental Combinations Corresponding Codes 
No Towel and Water Level 0 0,0 
No Towel and Water Level 1 0,1 
No Towel and Water Level 2 0,2 
No Towel and Water Level 3 0,3 
No Towel and Water Level 4 0,4 
Towel Position 1 and Water Level 0 1,0 
Towel Position 1 and Water Level 1 1,1 
Towel Position 1 and Water Level 2 1,2 
Towel Position 1 and Water Level 3 1,3 
Towel Position 1 and Water Level 4 1,4 
Towel Position 2 and Water Level 0 2,0 
Towel Position 2 and Water Level 1 2,1 
Towel Position 2 and Water Level 2 2,2 
Towel Position 2 and Water Level 3 2,3 
Towel Position 2 and Water Level 4 2,4 


Score. The students’ total scores on all three interactive tasks in this assessment and scores on the HF item 
were indicators of problem-solving ability. To explore the effectiveness of the classification of strategies, total scores 
on the assessment and scores on HF items were recorded and used for further analysis. Regarding scores on HF 
items consisting of five subtasks, the full credits for these five subtasks were 1, 8, 8, 1, and 6, respectively. Taking 
subtask 1 as an example, when one student selected “Towel Position 2” and “Water Level 2” (the correct answer), 
full credit 1 was given to this student. Additionally, the maximum possible score on the HF item was 24 based on 
the sum of full credits for five subtasks, and the maximum possible total test score was 76 according to the sum of 
full credits for all three interactive tasks in this assessment. 

Response Time. The time-based measures mainly include the execution time and total response time. Execu- 
tion time elapsed between the first and the last actions on the inquiry task in subtask 1 of the HF item, reflecting 
the time needed to perform a mental plan and execute response actions. Total response time is the total time 
spent on the HF item. 


Data Analysis 


Quantitative analyses, including descriptive statistics and ANOVA tests, were carried out to address the 
three research questions. Descriptive statistics were conducted to compare the aforementioned process-related 
measures across strategy groups. It should be noted that the data of the numbers of clicks for each experimental 
combination for each strategy group were generated into a two-dimensional array format via Python, and then 
the Matplotlib Library was applied to visualize data to heatmaps. To validate the classification of strategy groups, 
ANOVA tests on total test scores and HF item scores were conducted via the IBM Statistics SPSS 24.0 software. To 
explore the potential differences in the problem-solving processes among students using different strategies, the 
ANOVA tests were performed on the categorical variables (e.g., total response time, execution time). 
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Research Results 
Classification of Experimental Design Strategies 


When designing experiments to solve scientific problems, controlling the variables in the experiment is 
crucial to keep the other variable(s) constant and change only one variable under investigation across conditions. 
Additionally, even if a student can control variables scientifically, it does not mean that the student can design 
the entire experiment rigorously. According to the principles of controlling variables and designing experiments, 
students’ responses were classified into four strategies. This study has two variables: “Towel Position” and “Water 
Level.’Table 3 presents the descriptions of four strategies and their corresponding response actions. 


Table 3 
Descriptions of Four Strategies 


Strategies Descriptions of Strategies Corresponding Response Actions Representative Codes 


(1) Be able to keep the variable “Towel Position” constant —_[1, 0], (1, 1], [1, 2], [1, 3], [1, 4]; 
and collect five sets of “Water Level” data, and at least 2, 0], [2, 1], (2, 2], [2, 3], [2, 4] 


Scientific and Able to scientifically control two sets of “Towel Position” can be controlled 

rigorous variables and design rigorous 

(SR) experiments (2) Be able to keep the variable “Water Level” constant ——_{0, 0], [1, 0], [2, OJ; [0, 1], [1, 
and collect three sets of “Towel Position” data, and at 1], [2, 1] 


least two sets of “Water Level” can be controlled 


(1) Be able to keep the variable “Towel Position” constant —[2, 0], [2, 1], [2, 2], [2, 3], [2, 4] 
and collect five sets of “Water Level” data, but only one 


Scientific and Able to scientifically control “Towel Position” can be controlled 
less rigorous variables and design less rigorous 
(SLR) experiments (2) Be able to keep the variable “Water Level’ constant —_[0, 0], [1, 0], (2, 0] 


and collect three sets of “Towel Position” data, but only 
one “Water Level” can be controlled 


(1) Be able to keep the variable “Towel Position” constant, [1, 1], [1, 2], [1, 3], [1, 4] 
but collect incomplete data (only four or three sets of 


Able to control variables “Water Level” data) 
Incomplete nee 
(IN) but design incomplete and 
nonrigorous experiments (2) Be able to keep the variable “Water Level” constant, (0, 2], [1, 2] 
but collect incomplete data (only two sets of “Towel 
Position” data) 
Chaotic Unable to control variables and Unable to control variables, and collect data chaotically 1, 2], [0, 0], [2, 1] 
(CH) design chaotic experiments and irregularly 


To validate the classification, students’ total scores received on the assessment and on the HF item for each 
strategy group were compared. The results for total scores in Table 4 show that the students who adopted the SR 
strategy received the highest scores. In contrast, the students who used the CH strategy received the lowest scores, 
suggesting that the total scores declined with the emergence of the nonrigorousness of the experimental design 
when solving scientific problems. Additionally, for the SR strategy, the maximum value of total scores was 49.00, 
while the maximum value for the CH strategy was 34.00. Similar trends were found for scores on HF items. The 
students who applied the SR strategy got the highest scores on the HF item, while the ones who adopted the CH 
strategy received the lowest scores. Furthermore, the maximum value of scores on the HF item for the SR strategy 
was 20.00, while the CH strategy had the highest value of 16.00. 
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Table 4 


Descriptive Statistics of Total Scores and Scores on HF Items across Strategy Groups 


Scientific and Less 


Scientific and Rigorous Incomplete Chaotic 


Measure Rigorous 
M SD M SD M SD M SD 
Total Scores 23.79 10.24 19.43 10.07 17.17 9.62 10.21 7.03 
Scores on HF Item 7.23 4.12 5.86 4.02 5.03 4.13 2.52 2.89 


The differences in total scores and scores on HF items across strategy groups were further explored. Results 
of a one-way ANOVA on total scores showed that the means of the four strategies differed (F (3, 1512) = 124.573, 
p < .05). Games-Howell was utilized to run post hoc tests because the assumption of homogeneity of variance 
was violated according to the results of the Levene statistic in this study. Pairwise comparisons of the means us- 
ing Games-Howell tests indicated four significant comparisons: the mean score for the SR strategy (M = 23.79, 
SD = 10.24) was significantly (p < .05) larger than that for the SLR strategy (M = 19.43, SD = 10.07); the mean score 
for the SLR strategy (M = 19.43, SD = 10.07) was significantly (p < .05) larger than that for the IN strategy (VM = 17.17, 
SD = 9.62); the mean score for the IN strategy (M = 17.17, SD = 9.62) was significantly (p < .05) larger than that for 
the CH strategy (M = 10.21, SD = 7.03); and the mean score for the CH strategy (M = 10.21, SD = 7.03) was signifi- 
cantly (p < .05) smaller than that for the SR strategy (M = 23.79, SD = 10.24). Similarly, results for scores on HF item 
suggest that the means of the four strategies were unequal according to a one-way ANOVA, F (3, 1512) = 90.213, 
p <.05. Pairwise comparisons of the means using Games-Howell tests indicated four significant comparisons: the 
mean score for the SR strategy (M = 7.23, SD = 4.12) was significantly (p < .05) larger than that for the SLR strategy 
(M = 5.86, SD = 4.02); the mean score for the SLR strategy (M = 5.86, SD = 4.02) was significantly (p < .05) larger than 
that for the IN strategy (M = 5.03, SD = 4.13); the mean score for the IN strategy (M = 5.03, SD = 4.13) was significantly 
(p < .05) larger than that for the CH strategy (M =2.52, SD = 2.89); and the mean score for the CH strategy (M = 2.52, 
SD = 2.89) was significantly (p < .05) smaller than that for the SR strategy (M = 7.23, SD = 4.12). These results suggest 
that the total scores and scores on HF items were significantly different across strategy groups, implying that the 
classification of strategies can be confirmed experimentally. The strategies adopted when exploring and solving 
problems could predict problem-solving performance. 

Regarding the percentage of each strategy group in this study, the results showed that 18.5% of the problem 
solvers adopted the SR strategy. A total of 25.4% of the students applied the SLR strategy, and 31.5% adopted the 
IN strategy. A total of 24.6% of the students designed experiments chaotically and irregularly. Since the SR strategy 
is a more efficient way to solve scientific problems than other strategies, approximately 81.5% of eleventh-grade 
problem solvers need to improve their ability to design scientific and rigorous experiments to successfully solve 
scientific problems. 


Detailed Trajectory of Each Experimental Design Strategy 


The heatmaps of response actions reflecting the ways the students designed experiments and applied sci- 
entific knowledge were analyzed. The horizontal coordinate of the heatmap represents “Towel Position,’ and the 
vertical coordinate represents “Water Level.’ The numbers in each grid in the heatmaps represent the number of 
clicks on each experimental combination. As shown in Figure 2, according to the shades of color in the graphics, 
the students who applied the SR strategy mainly focused on controlling variables “Towel Position 2", “Towel Posi- 
tion 1” and “No Towel’, and especially collecting the data of combination of “Towel Position 2” and “Water Level 4”, 
suggesting that they cared about all key variables when designing experiments and were more curious about the 
combination of “Towel Position 2, Water Level 4” for lowering temperature; the students who adopted the SLR strat- 
egy paid more attention to control variable “Towel Position 2”, and especially collecting the data of combinations 
of “Towel Position 2, Water Level 4", and “Towel Position 2, Water Level 2”, implying that they might have believed 
“Towel Position 2” could play a better role in decreasing temperature than other towel positions and cared about 
the roles of “Water Level 4” and “Water Level 2”; the students who used the IN strategy mainly focused on control- 
ling variables “Towel Position 1” and “Towel Position 2”, collecting experimental data of combinations of “Towel 
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Position 2, Water Level 4’, “Towel Position 1, Water Level 2”, and “Towel Position 2, Water Level 2”, and ignored the 
data of “Water Level 0” compared to the SR and SLR strategies, suggesting an incomplete and ambiguous experi- 
mental design; the students who adopted the CH strategy paid attention to the combinations of “Towel Position 
2, Water Level 4’, “Towel Position 1, Water Level 2”, and “No Towel, Water Level 0”, while the design of experiments 
was completely discontinuous and chaotic. 


Figure 2 
Heatmaps of Response Actions for the Strategy Groups 
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Additionally, to further explore the concrete and detailed trajectories of experimental design for solving scien- 
tific problems for each strategy group, the top two most representative response action sequences for each strategy 
were recorded. Table 5 presents the top two most representative response action sequences and the frequency, 
proportion and examples of these sequences within each strategy group. Due to the various operations in the 
task for problem solvers, there were 140 sorts of response sequences among 280 students for the SR strategy, 338 
kinds of response sequences among 385 students for the SLR strategy, 409 kinds of response sequences among 
478 students for the IN strategy, and 125 sorts of response sequences among 373 students for the CH strategy. 
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Table 5 


Top Two Most Representative Response Sequences Across Strategy Groups and Their Frequency and Proportion Within Each 
Strategy Group 


zs 8 
Strategies ee ee Action 2 E = Examples 
c eg 
[0, O}', 0, 17, [0, 2], [0, 3), 0, 47, [1, OF, (4, 17, 1, 27, 1, 37, 
...[0, OJ, [0, 4], {0, 27, [0, 3], 0, 47, 1,47, 12, OY, ‘12, 17, (2, 27’, (2, 3], ‘12, 47’, 12, 47’, 12, 47’, 12, 4, 
(1, OF, (1,17, (1, 27, (1, 37, 11, 47, 79 2.2 862,21] 
Scientificand [OT 12, AY 2, 27, 12, SY Te AY... [[0, OY, 0, 47, 10, 2’, 0, 3), (0, 47, (1, Of, 1, 17, (1, 20, 1, 37, 
Rigorous [1,47 (2, OF, (2, 27, 12, 17, (2, 37, 12, 471 
(SR) (4, OF, (1, 11, (4, 27, (1, 37, (1, 47, (2, OF, (2, 17, (2, 20, 12, 37, 
TOF 10 1.27, 1 TTT, ay 057247, 12.4, 2.47] 
(2, OF, 2, 17, [2, 27, 2, 3), [2, 4... 
[0, 4], 10, 2), [0, 37, 10, 47, (4, OF, 4, 17, (4, 27, T1, 37, 14, 47, 
(2, OF, 2, 17, (2, 27, 2, 3), (2, 47, 12, 4, (2, 471 
([t, 4,01, 27, (1, 37, 11, 47, 12, OF, 12, 17, (2, 27, 12, 37, (2, 471 
hr, ...[2, OF, (2, 47, 2, 27, (2, 31, 12,47... 78 20.3 “F49 of, (0, OF, (0, 1P, 10, 3), 10.47, (1.47, 2.47, 2.4], 10.47, 
Less Rigorous 0, 47, (2, OY, (2, 11, 2, 27, (2, 3], 2, 47, 14, 47, 01, 27, 4, AY, 
(SLR) Tt, Of] 
(1, OF, (1, 17, (1, 27, ‘01, 37, 14, 47, 12, 47, (2, 47, 12, 471 
1, OF, (1, 47, 0,27, 11, 31, 0,47... 52 13.5 (10.0 (oP (h ay 37.47, DQ. Day, 
237, 12, 37, 2, 37] 
[(t, 41, 11, 27, (1, 37, Tt, 4, 2, 17, 12, 27, (2, 37, 12, 4, 12, 47, 
TT, 1.20, 11.37, 1,40 2 T,go ie ‘2, 47] 
‘ meen ({0, OF, (1, 4P, 11,27, 11, 3), 11,47, 2, 7, 2 21, 2 3), 1247, 
2.47, 12,47, 12, 471 
1,417, 11, 27, (4, 37, T, 47, (2, 41 
Tt, 1, (1, 20, 14, 30, 1, 40... 48 10.0 A a Ea eas Ee 
[ 0, O}, 0, OY, [4, 47, 11, 27, 14, 37, 11, 47, 14, 47, 12, 17, (2, 271 
Tl, 94 25.2 EMRIs aisetetinets Ubel] 
Chaotic (Tt, 27, 11, 27, 1, 27] 
(CH) nan aaA GaeIGL GC 
[2,4]... 81 21.7 UDO Petts We Tel 
([t, 4, 12, 27, (2, 47, 12, 47, (2, 471 


First, among the students who applied the SR strategy, the most representative sequence was exhibited by 
28.2% of the students and involved keeping the variable “Towel Position” (including No Towel, Towel Position 1, 
Towel Position 2, all three positions) constant and collecting five sets of “Water Level” data for each towel position. 
Another representative response sequence for the students who adopted the SR strategy involved keeping the 
variable “Towel Position” (including Towel Position 1 and Towel Position 2, two of the three positions) constant and 
collecting five sets of “Water Level” data for each towel position. 

Second, among the students who used the SLR strategy, the most representative sequence presented in Table 
5 involved keeping the variable “Towel Position” (including Towel Position 2 or Towel Position 1, only one of the three 
positions) constant and collecting five sets of “Water Level” data for each towel position. It is worth mentioning 
that the proportion of students who exhibited the actions of keeping the variable “Towel Position 2” constant and 
collecting five sets of “Water Level” data was more than that of the students who kept the variable “Towel Position 
1” constant, which is consistent with the results of the heatmap of response actions. 

Third, among the students who adopted the IN strategy, the most representative sequence exhibited by 17.2% 
of the students showed that although they were aware of keeping the variable “Towel Position” (including Towel 
Position 1 and Towel Position 2, two of the three positions) constant, they collected incomplete “Water Level” data 
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for each towel position. Another representative response sequence exhibited by 10.0% of the students involved 
keeping the variable “Towel Position” (including Towel Position 1, one of the three positions) constant but collect- 
ing incomplete data on “Water Level.’ 

Last, for the CH strategy group, the top two representative sequences were exhibited by 25.2% and 21.7% 
of the students, respectively, suggesting that the students paid attention to the experimental combinations of 
“Towel Position 1, Water Level 2” and “Towel Position 2, Water Level 4”. The findings are consistent with the results 
of the heatmap of response actions. The CH strategy group cannot design experiments to solve problems based 
on the principles of controlling variables. This group had no idea how to design experiments and merely solved 
problems irregularly and chaotically. 


Differences in Response Time across Experimental Design Strategies 


Table 6 shows that the students who adopted the SR strategy spent the most time-solving problems. In con- 
trast, the students who used the CH strategy spent the least time responding to tasks, suggesting that the students 
who applied rigorous and scientific strategies needed more time to solve scientific problems. 


Table 6 
Descriptive Statistics of Total Response and Execution Time Across Strategy Groups 


Scientific and Less 


Scientific and Rigorous Incomplete Chaotic 


Measure Rigorous 
M SD M SD M SD M SD 
Tolal response Wns: - £4956 309.46 428.85 270.13 348.09 273.64 231.78 228.54 
(seconds) 
Executions dite 71.66 46.12 59.75 50.02 41.98 34.07 21.04 34.01 


(seconds) 


Regarding the total response time, the results suggest that the means of response time for the four strategy 
groups were unequal according to a one-way ANOVA, F (3, 1471) = 71.603, p < .05. Pairwise comparisons of the means 
using Games-Howell tests indicated four significant comparisons: the mean time for the SR strategy (M = 532.26, SD = 
309.46) was significantly (p < .05) longer than that for the SLR strategy (M = 428.85, SD = 270.13); the mean time for the 
SLR strategy (M = 428.85, SD = 270.13) was significantly (p < .05) longer than that for the IN strategy (M = 348.09, SD = 
273.64); the mean time for the IN strategy (M = 348.09, SD = 273.64) was significantly (p < .05) longer than that for the 
CH strategy (M = 231.78, SD = 228.54); and the mean time for the CH strategy (M = 231.78, SD = 228.54) was significantly 
(p < .05) shorter than that for the SR strategy (M = 532.26, SD = 309.46). 

Similar trends were found for the execution time. The results suggest that the means of execution time for the 
four strategy groups were unequal according to a one-way ANOVA, F (3, 1388) = 89.486, p < .05. Pairwise comparisons 
of the means using Games-Howell tests indicated four significant comparisons: the mean time for the SR strategy (M = 
71.66, SD = 46.12) was significantly (p < .05) longer than that for the SLR strategy (VM = 59.75, SD = 50.02); the mean time 
for the SLR strategy (M = 59.75, SD = 50.02) was significantly (p < .05) longer than that for the IN strategy (M = 41.98, SD 
= 34.07); the mean time for the IN strategy (M = 41.98, SD = 34.07) was significantly (p < .05) longer than that for the CH 
strategy (M = 21.04, SD = 34.01); and the mean time for the CH strategy (M = 21.04, SD = 34.01) was significantly (p < 
.05) shorter than that for the SR strategy (M = 71.66, SD = 46.12). 


Discussion 
Different Ways of Designing Experiments and Applying Scientific Knowledge Across Strategy Groups 
The trajectories of the four strategies in this study reflected various ways of designing experiments and applying 


scientific knowledge when solving scientific problems. Regarding the ways of designing experiments, the SR group that 
achieved the highest total scores paid attention to all variables and knew how to scientifically control variables and 
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design rigorous experiments to explore the best conditions for lowering the temperature. The SLR group was also aware 
of controlling variables scientifically and mainly paid attention to the variables they cared about; therefore, the design 
of experiments was not rigorous and systematic enough. The IN group knew the variables needed to be controlled 
when designing experiments, while the experimental data for exploring the best condition for lowering temperature 
were not valid enough because the design of experiments was incomplete and nonrigorous. The CH group, which is 
also a low-performing group according to their total score, was not aware of controlling variables when designing ex- 
periments and designed experiments in a completely chaotic and irregular way. To explore the condition for the best 
refrigeration effect, students need to combine various variable conditions (“Towel Position” and “Water Level”) to set up 
experimental combinations for comparison. The process of setting up experimental combinations requires a good deal 
of cognitive resources for controlling variables and designing experiments. The need for abundant cognitive resources 
to design experiments is consistent with the literature (Gong et al., 2020). The patterns of designing experiments to solve 
complex scientific problems suggested that successful problem solvers were generally inclined to explore all possible 
experimental combinations and design experiments scientifically and rigorously. 

Regarding the application of scientific knowledge, the SR group paid more attention to the combination of “Towel 
Position 2, Water Level 4”, and the SLR group cared more about the combinations of “Towel Position 2, Water Level 4”and 
“Towel Position 2, Water Level 2” Therefore, understanding the roles of water levels 2 and 4 was crucial for the students 
to design experiments. Water level 2 is related to the principle of evaporation and heat absorption. The towel absorbs 
water in the pot, and then the water evaporates and absorbs heat from the bowl. The process of evaporation and heat 
absorption plays a vital role in the refrigeration effect. Water level 4 is relevant to another scientific principle that heat 
transfer occurs when the water in the pot submerges the bowl. Due to heat transfer, water submerging in the bowl 
would transfer the heat of the water to the bowl to reduce the refrigeration effect. Understanding the aforementioned 
principles is useful for students to design experiments. The findings showed that many attempts were made to explore 
what would happen when water submerges the bowl (water level 4) and does not submerge the bowl (water level 2). 
The SR group students made many more attempts to discover the roles of water level 4 than level 2, implying that the 
SR group might be aware of the scientific principle of evaporation and heat absorption but not heat transfer. In contrast, 
the SLR group cared about water levels 4 and 2, suggesting that the SLR group might be confused about the principle 
of evaporation and heat absorption and the principle of heat transfer. The results indicated that the relevant scientific 
knowledge and principles in students’ minds might affect the use of strategies and the ways they design experiments 
when solving scientific problems. Successful problem solvers have systematic and structured knowledge, while un- 
successful problem solvers tend to have less scientific and unorganized knowledge (Zajchowski & Martin, 1993). Our 
findings are consistent with some research showing that conceptual knowledge is basic for students to solve scientific 
problems and is a predictor of problem-solving competency (Friege & Lind, 2006). 


Sufficient Time for Solving Complicated Scientific Problems 


Results show that the students who used more scientific and rigorous strategies soent more time seeking solu- 
tions when solving scientific problems. Since the scientific problems that occur in scenario-based tasks and daily life 
are generally complicated and authentic, successful problem solvers usually need to spend more time making plans 
and finding solutions to better solve the problems in the processes of dealing with difficulties (Cowie, 2015; Gong et al., 
2020). The previous findings are consistent with the total response time and execution time results for different strategy 
groups in this study. That means that to design a scientific and rigorous experiment to successfully solve a scientific 
problem, ample time might be used to reach the goal. When solving problems, while both successful and unsuccessful 
problem solvers consider a given problem situation in terms of the problem's objectives, successful problem solvers 
tend to spend sufficient time making assumptions and mapping the problem situation to an appropriate theoretical 
model by retrieving valid representations obtained from their larger and better-organized knowledge base (Yerushalmi 
& Eylon, 2015). Moreover, successful problem solvers may also spend more time than unsuccessful problem solvers 
reflecting on their former decisions and revising their choices accordingly (Yerushalmi & Eylon, 2015; Jiang et al., 2021). 

An explanation for the relationships between time on task and task success may be provided by dual process- 
ing theory, which suggests a dynamic interaction of automatic and controlled mental processing (Schneider & Chein, 
2003). Automatic processes that are fast do not require active control, whereas controlled processes that last longer 
than automatic processes require attentional control. Researchers argue that response time on a task is dependent on 
the relative degree of automatic versus controlled processing required by the task (Goldhammer et al., 2014). Due to 
the complex processes of problem solving containing knowledge acquisition and the application of this knowledge to 
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generate solutions, scientific problem solving must rely on controlled processing to a substantial degree, especially in 
some difficult tasks. Previous research has indicated that task difficulty affects the relationship between time on task 
and task success (Goldhammer et al., 2014). It can be assumed that easy tasks require essentially automatic processing 
to complete, whereas difficult tasks require a higher level of controlled processing. Although successful problem solvers 
proceed more quickly from information identification to integration, they need to invest sufficient time in seeking solu- 
tions rather than in low-level information processing. Accordingly, time spent on problem-solving tasks may increase 
with task difficulty. 


Conclusions and Implications 


The purpose of the present study was to explore the eleventh-grade Chinese students’ problem-solving strate- 
gies and the differences among strategy groups by analyzing process data including sequences of students’ response 
actions and response time. Results showed that students mainly adopted four strategies for designing experiments to 
solve complicated scientific problems. Successful problem solvers were generally inclined to explore all possibilities 
of experimental combinations and design experiments scientifically and rigorously based on the relevant scientific 
principles. There were significant differences in the response time used for solving problems among different strategy 
groups. The students who adopted more scientific and rigorous strategies spent more time designing plans and seeking 
solutions to solve complex scientific problems. 

The results enrich the literature on using process data obtained from interactive items to address theoretical issues 
in educational assessment and provide actionable feedback for teaching and learning the skills required in scientific 
problem-solving tasks. First, students need to be guided and instructed to be aware of the principles of designing con- 
trolled experiments when solving scientific problems. Second, students’ understanding of the principle of evaporation 
and heat absorption affected how they designed experiments to solve problems. Science teachers should help students 
construct a systematic and organized knowledge structure. Third, science teachers should give students sufficient time 
and ample opportunities to adequately participate in problem-solving activities and make elaborate plans for seeking 
solutions to cultivate students’ scientific problem-solving ability. The length of time for carrying out problem-solving 
activities should be based on the students’ academic performance (e.g., length of time to fully design and complete the 
experiment) and the difficulty of the tasks. 

One limitation of this study is that the conclusions might not be applicable to all Chinese students since the data 
were collected from students in southern China through a convenience sampling technique with a limited number of 
sample students. Considerably more research should be conducted to discover patterns of students’ problem-solving 
strategies in a large-scale assessment in other economic or cultural contexts. Additionally, more general conclusions 
about science could not be reached in this study since the analysis was performed only on one subtask isolated from the 
set of 17 subtasks in total and the subtask merely belonged to the domain of Physics. More subtasks covering all science- 
related domains should be analyzed in the future to complement and enrich the findings on students’ problem-solving 
strategies. Moreover, the classification of strategies for solving scientific problems was mainly based on the analysis of 
process data reflecting students’ response actions. In a follow-up study, it would be helpful to further explore the specific 
descriptions of each strategy group using qualitative analysis, such as think-aloud protocol analysis and interviews. 
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